search
attachments
weblink
advanced
Overview
Content Tools
Work Completed
- KISTI
- Lucene index is now written to disk. One additional tweak (noted in the source code) to this would be for the server to have a properties file to specify where the lucene index should be written or the index might fail to get created if the folder permissions are not correct.
- I started updating search, but haven't tested it yet. Most of the code is there, it just needs to be tested and checked in.
- I created another eAIRS parameter tool (based on the first) that implements CyberintegratorToolTupelo interface. This only required minor code changes from the existing tool, but the new tool is provided access to the Tupelo context and can write metadata, if needed. The interface differences between CyberintegratorToolTupelo and CyberintegratorToolFile are minor.
- Built and deployed eAIRS locally for testing since we were having issues with correct versions of interfaces being built. It turns out that you have to be careful building with Maven because if you have old versions of a project already built, it might include the old jar instead of the new one (build order matters). Wiping out local copies of the projects forces the build order to be done properly.
- Attended Monday Staff meeting via skype
- Attended Tupelo meeting in Champaign.
Work Planned
- KISTI
- Continue working on KISTI eAIRS-web. The lucene index should be written to disk and should be updated when a user submits a workflow. The search view needs to be updated to limit search by user or by all users. I also plan to create a tupelo-tool version of the eAIRS parameters tool to make it easier to add metadata from within the tool.
- Attend 10am staff meeting on November 7th
- Drive to Champaign to attend Tupelo meeting on November 11th.
Comments
This week went as planned.
Work Completed
- KISTI
- Revised KISTI final report on Saturday October 29th and sent my revisions to Jong for final adjustments before submission.
- Attended Staff meeting
- Split Browse and Search into separate views. I also added a temporary Lucene index that gets built when eAIRS-web starts to provide search capabilities for published workflows. This still needs to be expanded so that when a new workflow is published it needs to get added to the index. We also should have something more permanent so the lucene index doesn't always need to get regenerated.
- Updated Simulation Monitoring view's download button to call the REST endpoing that Jong created for downloading workflow outputs as a zip. Luigi helped correct some minor bugs in both the front-end and back-end code.
- Tested eAIRS-web to make sure everything was working for the KISTI demo. Once this was complete, I recorded an end-to-end screencast highlighting the features implemented so far.
Work Planned
- KISTI
- eAIRS-web Search and Browse need to be split into separate views and the UI and backend still need work. For the demo we should add a lucene index that gets built on startup based on what is found in the repository.
- Look into a bug that causes the last run workflow to be seen in the Browse published workflow views.
- Update Simulation Monitoring view download button to call out to the REST service to download the output data for that workflow
- Test eAIRS-web functionality and Record video demo of eAIRS-web
- Attend Staff meeting at 10am on October 31.
- Revise KISTI final report over the weekend (October 29th)
Comments
This week went as planned.
Work Completed
- KISTI
- Drove to Champaign on Monday October 24 for all day work session
- Worked on Simulation monitoring table to display creation date to better determine which run was launched when. I also worked on the search page and adding code to use Lucene for search. More needs to be done to start storing information in Lucene. Most of the framework is in place to use Lucene for search, it just needs a lucene index to call out for searching simulations (and other things that we will want to search for in the repository).
- Fixed a bug in the eAIRS tools where the field type for the workflow script should have been application/xml not text/plain
- Worked with Luigi on testing the publishing and submitting of workflows using the PAW submission code
- Created a tutorial guide for creating a new Cyberintegrator workflow and exporting/importing it into Cyberintegrator
Work Planned
- KISTI
- Drive to Champaign for all day work session on October 24, 2011
- Continue working on eAIRS-web UI
- Work with Luigi on submitting a workflow using PAW submission code
Comments
This week went mostly as planned. Next week we will continue to pull together the eAIRS web application for release.
Work Completed
- KISTI
- Reviewed and updated Ember resource request document.
- Updated workflow scripts for Ranger (2d and 3d) and tested them on Ranger. I found that the KISTI.ncsa REST service wasn't used the latest Gondola with the new package name so that had to be updated. After it was updated, the scripts ran successfully.
- Updated the eAIRS tools with the plug-ins containing the new package name. I also updated the eAIRS tools PTPFlow plug-in dependencies to remove the org.apache.commons plug-in. That plug-in bundled several apache libraries and could cause classloading issues. The updated plug-ins are on ncsa-orbit. I rebuilt the tools and tested them both locally and on the KISTI CI server so that the updated tools and scripts would be on the remote resource for others to use/test.
- Continued working on the eAIRS UI. I added some tweaks to the workflow monitoring UI and started working on the Search UI. This will continue next week.
Work Planned
- KISTI
- Work on eAIRS-web User Interface
- Update and Test Regex for job submission on Ranger. Also, eAIRS tools will need to be updated with the latest package change to Gondola since the JAXB class packages changed with the move to opensource
- Update eAIRS tools dependencies. The PTPFlow plug-ins that are depended on should be updated to remove org.apache.commons and checked into ncsa-orbit.
- Review Ember resources request document
Comments
This week went as planned. I took a personal day on Monday October 10th.
Work Completed
- KISTI
- Worked on job monitoring UI. I reviewed the cell widgets that Luigi suggested for the search simulation mockup and determined that cell widgets would also be a good fit for the job monitoring UI. I updated the WorkflowService API for requesting workflows from Tupelo so that I could retrieve WorkflowBeans through an API call and have the Job monitoring page update the status of the job with simple paging (only displays 10 rows at a time). We can tweak this later if we would prefer more to display. More work needs to be done so we can (for example) only show the users jobs/workflows.
- Tested the updated REGEX provided by Jong for our job submissions
- Discussed org.apache.commons and removing that dependency from the Gondola code. That plugin wraps multiple apache libraries, which is not good practice and can cause runtime errors if something else loads one of the libraries that is bundled.
- Worked with Jong on exporting the Gondola product for multiple platforms.
- Luigi helped me with some issues I was having with the generated Async not getting updated by using Maven > Clean. This cleared up some issues I was having with building some of the Aysnc methods I needed for my job monitoring UI.
Work Planned
- KISTI
- Work on the workflow monitoring UI for eAIRS and possibly the simulation search UI mocked up on slide 15 of the eAIRS mockup
- Work on other parts of eAIRS as required (e.g. Gondola)
- Attend KISTI staff meeting (Mondays, 10am)
Comments
This week went as planned. I took a half day on Friday October 7th.
Work Completed
- KISTI
- Tested Gondola REST service and Cyberintegrator Server on kisti.ncsa.illinois.edu. I updated the Gondola Server to read the logging level from a properties file and used hard coded logging as the fall-back if there is a problem reading a properties file or no properties file provided. This allows administrators to set the logging level for each package at runtime.
- I submitted a help request to XSEDE for obtaining a password for my Blacklight account since none was provided. After getting my credentials, I successfully compiled the eAIRS 3d code and submitted a job. I then worked with Jong to updated the REGEX for the script file so the Gondola client can obtain job information.
- Luigi finished some code for publishing workflow beans and I briefly reviewed it
- Updated paw-authentication and went through the steps to setup a Mysql database locally to log into paw-authentication. This will be needed for getting credentials and running the published workflows
- Started working on the UI for job monitoring. I converted some of my earlier code to the Model-view-presenter (MVP) methodology. I need to finish the front end and then start adding in real data from the REST service.
- Assisted Dr. Nam in getting Gondola running on his machine from his Eclipse workspace
- General
- Submitted my completed Conflict of Interest form to HR
Work Planned
- KISTI
- Test updated REST service and updated CI server with eAIRS tools.
- Update Gondola server to read logging level and package names from a properties file at runtime
- Obtain credentials for blacklight account and test running eAIRS tools on blacklight
- Start working on more of the UI parts with Luigi and convert current mockups to the MVP methodology.
- General
- Submit conflict of interest form to HR
Comments
This week went as planned. Next week we should be able to publish a workflow and try running it using PAW.
Work Completed
- KISTI
- Attended Monday staff meeting.
- Worked on Gondola REST service. I added CET logging plugins to the headless RCP application and added an slf4j to log4j bridge library that now allows us to capture all logging statements. Using the CET logging, we can now control what log level gets written to the log file for each package. I was not able to reproduce an occasional bug I saw with the KISTI gondola service, but once we start using my updated code we should be able to see what is going on when the error happens. I also noted that we will need to improve this service to use the Tupelo Repository for getting the latest user credentials. The current implementation will be prone to failure once the clients cert expires and the job hasn't ran yet because it stores the cert locally inside it's own repository. It would be better if it went out and got the latest cert from Tupelo.
- Finished up the 3d solver testing. Everything is operating normally and is checked into SVN. The next step is to start wiring this up with the Web interface to setup and launch jobs.
- Started transitioning my Desktop files to my laptop. It took some time to get the laptop dock to work properly with Linux. I spent most of Friday installing the system and retrieving all required applications. The laptop is considerably faster than my aging Desktop and will improve efficiency (Maven compile/installs would grind my desktop to a halt until it finished, it's barely noticeable on the laptop). The laptop is dual booted win7/ubuntu 11.04.
- Provided Yong Wook some advice on getting started with transitioning his paw-authentication code to the Model-view-presenter design paradigm.
Work Planned
- KISTI
- Work on improving the Gondola REST service. There is an occasional bug that shows up, but no logging is being captured because of the different logging systems among the libraries used
- Work on finishing up the 3D solver tools and test everything using KISTI-tupelo repository and KISTI-CI server
- Transition desktop machine to laptop and installed 64 bit Ubuntu on laptop
- Attend Monday Staff meeting (10am)
Comments
This week went as planned. It took a little longer than anticipated to setup my laptop as my primary workstation since I had to resolve issues with Ubuntu recognizing the dock properly. It took a bios update and some bios tweaking to resolve the problem.
Work Completed
- KISTI
- Compiled and tested eAIRS 3D solver. There were some issues getting the code to compile and then some runtime errors due to the environment settings on XSEDE's Ranger resource. I setup my desktop system to compile and run MPI code so I could test runtime errors more quickly by avoiding the gueue time. I sent questions to KISTI involving the code and output files and used the Ranger online manual to change my environment settings and resolved the issues I had. I created a new script for the 3d workflow that loads the correct environment before executing the solver. The solver is tested and a new 3d tool was created and checked into SVN since the 3d solver has different outputs than the 2d solver. It took more time than anticipated to compile and successfully run the 3d solver, but everything is working and tested.
- Reviewed in more detail the GWT material on the MVP design pattern and sent some questions to Luigi.
- Attended staff meeting (9-12-11) via skype
- General
- Prepared short writeup summarizing TRECC activities
Work Planned
- KISTI
- Compile and test eAIRS 3D solver received from KISTI
- Create new Cyberintegrator tool to execute eAIRS solver
- Attend KISTI staff meeting (9-12-11)
- General
- Prepare short writeup for Terry to summarize TRECC activities
Comments
This week went as planned. I took half of a personal day on Friday September 16, 2011.
Work Completed
- Cyberintegrator
- Verified my classloader change with Rob, tested it and checked in my changes. We rebuilt Cyberintegrator server and updated our CI server installation at kisti.ncsa.illinois.edu with the new build
- KISTI
- Fixed the issue Yong was having with getting credentials associated with a user. There was a missing Triple in the Utility class
- Finished updating the eAIRS tools to look for credentials associated with the step or with the user. The first check is for credentials associated with the step, it then falls back on trying to find a credential associated with the creator. We will need something to associate the users credential with the workflow steps so they can access HPC resources. I also checked in a README file with the tools that will help users export the tools as jar files and import them into Cyberintegrator. A workflow for the tool is also checked into the repository. All of these tools were tested with the new CI server to verify correctness
- General
- Attended all-staff meeting on Thursday
Work Planned
- Cyberintegrator
- Verify new code for DisposeClassLoader with Rob and update CI Server. The server will need to be rebuilt with Jenkins and redeployed.
- KISTI
- Fix bug in x509 certificate util where associations returns the empty set
- Finish updating eAIRS tools to use the new REST client and obtain the credential from Tupelo either by searching for an association with the workflow step or the creator
- General
- Attend all staff meeting
Comments
This week went as planned. September 5th was a holiday and I took half of a personal day on September 7th and 8th.
Work Completed
- KISTI
- Updated eAIRS tool to use the new PTPFlow-lite version and REST endpoing to submitting jobs. KNSG-107. This took some time due to classloading issues that were slow to track down between the REST client library and also with some of the JAXB classes used by PTPFlow-lite. The primary issues with PTPFlow-lite was fixed by specifying directly the Factory class and it's full path (found online in a forum) for issue encounted when mixing Eclipse Plug-in code and Java Standalone clients.
- I am in the middle of testing the end to end workflow. My final changes are to update the basic eAIRS script dynamically with information such as which machine to run on, the machine's gridftp URL, etc. Once these tests are complete we can begin working with the front-end client code to setup/launch this workflow.
- Started looking at Yong Wook's issue with getting the associated certificate, the bean util's look correct. I will continue to look into this.
- Attended KISTI staff meeting
- Attended Luigi's GWT-MVP tutorial
- Cyberintegrator
- Added an unimplemented findResources(...) method that was not part of Cyberintegrator's DisposeClassLoader. This caused a classloading issue for the REST library eAIRS is using because it's class loading utility class was calling ClassLoader.getResources(...), which would return null since it is unimplemented. I need to verify my changes with Rob before committing it back to the Cyberintegrator.
Work Planned
- KISTI
- Update eAIRS tools to use PTPFlow-lite client for submitting jobs.
- Attend KISTI staff meeting on Monday at 10am.
- Attend Tutorial on GWT-MVP
- Cyberintegrator
- None
Comments
This week went mostly as planned. It took longer to fix classloading issues than anticipated, but they are now resolved and the final code updates are being done for the eAIRS tools. This should be finished early this week so we can begin connecting it to the UI parts. The final issue will be getting the x509 certificate associated with the workflow step so the step can get the correct cert without logging into MyProxy or relying on the user's tupelo username to get at the certificate (in case they are logged out and no cert is associated with their Username).
Work Completed
- KISTI
- Visited Champaign on Tuesday August 23, 2011 to attend Eclipse PTP User meeting and tutorial. The tutorial session was very helpful. I successfully setup and ran a sample MPI application on Lincoln.
- Tested REST API with test scripts , including a test script that ran eAIRS on Ranger. The next step is to re-work the Cyberintegrator eAIRS tool to use the REST service instead of the PTPFlow RMI Client
- Continued working on monitoring UI and simulation details UI pieces. I also looked over the UIBinder/MVP tutorial links that Luigi provided.
- Attended staff meeting
Work Planned
- KISTI
- Attend Eclipse PTP User meeting on August 23, 2011
- Work on eAIRS/PAW UI
- Test REST API and start converting eAIRS tool to use REST API instead of RMI Client
- Attend weekly staff meeting (10am Mondays)
Comments
This week went as planned. Next week I will continue working on the eAIRS/PAW UI pieces and update the eAIRS tool to use the REST service.
Work Completed
- KISTI
- I setup my development environment with the new Eclipse Indigo release and Maven plugin. It took some time to resolve issues I had getting Maven to update the PAW plugins properly and to build the PAW modules.
- Looked over KISTI REST API
- Checked out and looked at the PTPFlow-light client code (Gondola). I suggested a dependency change to Al to remove unnecessary plug-ins that could conflict. Most of the libraries wrapped inside Al's org.apache.commons plugin are now bundled with Eclipse and can be automatically picked up.
- I updated paw-authentication to include a new dependency (to include tupelo-client) and I added code for storing the x509 certificates in Tupelo with my X509CertificateBean. One issue that needs to be addressed is how does paw-authentication get connected to a Tupelo context. In order to associate the x509 with a specific user, we need to have that user first sign into tupelo with a U/P or support some kind of single sign-on between Tupelo and MyProxy. Once this is addressed, we can use my code to store the x509.
- I started working on the GWT layout for the job monitoring UI based on slide-22 of the EDISON v2.2.1 document provided by KISTI. This is still a work in progress as I get more familiar with the available GWT widgets and UI binder.
- Attended KISTI staff meeting (Monday 10am).
Work Planned
- KISTI
- Add code to paw-authentication module to store x509 certificate in tupelo
- Start working on UI development for eAIRS/PAW
- Attend KISTI Staff meeting, Monday - 10am.
Comments
This week went as planned. Next week I will be in Champaign (August 23rd) to attend the Eclipse PTP user meeting.
Work Completed
- KISTI
- Worked on installing and getting ELF running on Ranger. I am working through qsub submission issues with help from Al. Ranger is setup differently than other machines ELF is running on.
- Tested PTPFlow client login for Yong Wook. I was not receiving the cert expired warning so I provided suggestions on checking the grid certificate. The issue was resolved.
- I did not get to updating PAW Authentication. I will need guidance on setting up my dev environment and running the current code. When I downloaded the source code I was getting build errors.
- Cyberintegrator
- None
- Medici
- None
Work Planned
- KISTI
- Install and Test ELF on Ranger HPC
- Update PAW authentication to store certificates with our X509CertificateBean
- Cyberintegrator
- None
- Medici
- None
Comments
This week went mostly as planned. It is taking longer than anticipated to setup ELF on the Ranger system.
Work Completed
- General
- Attended all-staff meeting via phone call in.
- KISTI
- KNSG-106, staging tool. The current version is a command line tool, but it would be better as a Java tool so it can take advantage of Cyberintegrators facilities for providing a Context because it needs access to the grid certificate that will be stored in tupelo. Otherwise, we would need some other way to get the certificate to the command line tool. The conversion is finished.
- KNSG-109, checked in changes to the X509CertificateBean and bean utility. It is now ready for use with the login code to store x509's by user.
- KNSG-107, made changes to the PTPFlow eAIRS Java tool to obtain the user from the workflow step based on discussions I had with Rob.
- Attended KISTI project staff meeting.
- Cyberintegrator
- Found and fixed a bug in Cyberintegrator server, CBI-379. There was an extra slash in the URL that caused it to fail to connect. This was found while testing the KISTI workflow launched on a remote Cyberintegrator server. The fix is checked into SVN.
- Medici
- None
Work Planned
- KISTI
- Attend August 1st Staff meeting from 10am - 11am
- Work on finishing the bean for storing X509 certficates, KNSG-109
- Work on tool for staging files, KNSG-106.
- Cyberintegrator
- Medici
- None
Comments
This week went as planned. I took a personal day on Thursday August 4th and half of a personal day on Friday August 5th.
Work Completed
- KISTI
- Worked on KNSG-107. I ran into issues with class loading between what Cyberintegrator had loaded and what was bundled in my PTPFlow tool. I finally narrowed it down to a difference between the commons logging in org.apache.commons from PTPFlow and the commons logging that comes bundled with Eclipse. org.apache.commons is mostly outdated and for all PTPFlow plugins relying on it for logging, this dependency can be safely removed. I intend to ask Al's permission to remove the depedendency on org.apache.commons since it is no longer needed. I got as far as running my PTPFlow tool as a CyberintegratorToolFile, which does not provide a Context to the tool. I am now working on Converting the tool to CyberintegratorToolTupelo (nearly finished) so we can get a certificate from an X509CertificateBean.
- Worked on KNSG-108. I got some feedback from Rob and I am working on integrating that feedback into the bean and it's utility class. This should be tested and finished soon.
- Worked on KNSG-109. Instead of using a command line tool where you could specify each parameter when adding the tool to Cyberintegrator (initial concept), I determined it would be easier to create a ModelParameter java tool where the UI would be auto-built by Cyberintegrator based on the class definition. The output of this tool is a parameter file. Everything is checked into svn at in the package edu.illinois.ncsa.knsg.eairs.tools
- Cyberintegrator
- None
- Medici
- None
Work Planned
- KISTI
- Cyberintegrator
- None
- Medici
- None
Comments
This week went mostly as planned. It took me longer to progress on the class loading issue because there were no error messages.