• Basic description of each one
• When to use one over the other
• Examples using Clowder docs and new landing page

2020-01-22 Meeting notes

• Basic knowledge of docker required
• Basic commands
• Helm charts
• ???
• An old slide deck
• An older slide deck

• Investigate ingress controller technologies
• Pick best practices and guideline to pick a right technology
• Possibly pros and cons of each option?

• Keycloak vs CI-Logon
• OAuth 2, OIDC
• User (identity) management: NCSA service or not?

• Active software projects that could realistically be used on future projects
• Expertise in 3rd Party Software
• By department? By people? By software?

• One platform to transition from Data Science experiments
  to a full web app
• Lessons learned thus far and open problems

• I'm doing this right now (Nest and its projects: Knoweng, Omix, etc). 
• Also adopted Conda for python and R dependencies
  INSIDE our docker containers.
• Plenty of pain points along the way if anyone wants to hear about it.

• Visual Analytics Group wants to do this in the next few months
• Any lessons learned would be helpful

• Which kind of project goes better at NCSA? 
• Hosting a service vs. instances of open source software run by others?

• Software Directorate seems to be moving to standardize on containers 
  with orchestration.
• NCSA still puts Condor or similar batch management software on the
  cutting edge hardware. Can't even run a docker container.
• This seems like a huge problem to me but maybe I'm the only one.

• Onboarding FT hires, student hires, collaborators
• Writing proposals
• Kicking off new projects
• Releasing open-source software
• Capturing UI/UX requirements, developing concepts, and arriving at final designs
• Translating UI/UX requirements into frontend implementations
• Lots more?

• "frameworkless" or cross-framework interactive HTML widgets

• syncing live data across multiple browser windows
• An old slide deck

• what are we using? what do we like?

• service desks/user-facing ticketing
• knowledge bases

• custom plugin capability of jupyterlab
• custom authenticator of jupyterhub
• How to deploy and mantain a Jupyter Hub instance for a community

• re-usable react components (e.g. login)
• other frontend libraries (angular?)
• visualizations?

• A look at the less technical side of software development patterns
• Requirements-gathering, ticket composition, use cases, test plans, and CHANGELOGs
• Not JIRA-specific, but lots of JIRA examples
• An old slide deck

• How to create cluster of Geoserver for better scalabilty 
• How to make Pods of Geoserver on Kube
• How to improve Geoserver performance

• How to manage Raster data on PostGIS (Postgresql)
• What's the best practices with different options
• Examples of geospatial query over raster data on PostGIS

• Understands the differences (pro/con) between OpenLayers vs Leaflet
• Survey and review the JS libraries (i.e. react, bootstrap, etc.) working with OpenLayers or Leaflet 
• What's the best practices with examples 
• What's best practices working with Geoservers (WMS/WFS) services

• What's the options regarding parallelization of geospatial analysis (vector/raster) on HPC (CPU based) (MPI), HPC (GPU based), Cloud (i.e. dockers, kube)
• What's the best practices or recommendation?

• TypeScript vs Flow
• When is TypeScript better than plain ES6
• Some projects at NCSA are currently using TypeScript
• Typescript on the backend (NestJS, Deno, tsc or ts-node w/ Node/Next/Ember/Fastify, etc.)

• File types: json, tif,shp, csv etc. Mostly files uploaded from webservices
• Pro & Cons

• Roles, Groups, Users giving other users access to their data.
• Application level? database builtins? Are there libraries?
• Especially when querying large sets of data for analysis (but don't have permissions to all data of a type).

• ipywidgets
• javascript-based solutions (code-sharing with web apps?)
• Incorporating notebooks in web apps (e.g., https://blog.jupyter.org/and-voil%C3%A0-f6a2c08a4a93)
• •nbclient, a very lightweight python API for executing notebooks - https://nbclient.readthedocs.io
  •Papermill, a tool for parameterizing and executing Jupyter Notebooks - https://papermill.readthedocs.io
  •Jupytext, a converter between notebooks and code and vice versa - https://jupytext.readthedocs.io

Identifying the options and advantages of tools related to workflow execution, especially related to deployment on public clouds

• Workflow Languages: Common Workflow Language (CWL); Workflow Descriptor Language (WDL)
• Execution Engines: Rabix; Toil; Cromwell; cwltool; Data Wolf; Apache Airflow; Parsl;
• Workflow-compatible Cloud Compute Platforms: Seven Bridges CGC; Terra/AnVIL;  
• Workflow Language Support Tools: Dockstore registry, CWLviewer, argparse2tool  
• Support communities: GA4GH Cloud Workstream 

• When to use GitHub vs NCSA opensource
• What Github features to use (or not)
• GitHub best practices
• Plugins for automatic interaction
• Personal experiences using GitHub

2021-01-27 Meeting Notes

• Benefits, price model, availability

• When should we use it
• How to scale
• Integration with existing software

• Keeping your secret safe in git and github

• Started as a request from INCORE team member

• what is the simplest way to expose a flask service to the world
• static web pages
• how to auto deploy containers, static web pages, services 
  • start stop docker container
  • pull git repository

• when is a project big enough to use helm chart
• best practices, example, tutorials with helm charts 
• standard repositories and practices

• single sign on - Oauth vs cilogon
• external directory - Ldap
  • identity.ncsa.illinois.edu
• manage local users
• (Jupyter hub authentication)
• GitHub Auth flow and scope

• pandas, R data frames
  • performance difference between APIs
• numpy arrays
• hdf5 serialization
• dask dataframes
• fits (astronomy)
• netcdf
• Apache Spark

9/22/2021

https://acrobat.adobe.com/link/track?uri=urn:aaid:scds:US:5544745d-4f14-498b-92e8-aa042fea3377#pageNum=1

• simplifying nested callbacks

• for the kind of presentations that we give, what are best practices
• conferences vs outreach opportunities like engineering open house
• how to structure a talk
• verbose slides vs diagrams
• different audiences, technical, non-technical
• how detailed
• how to explain abstract ideas (diagrams, examples, code)

• Postgres/Supabase.io, Firebase, Python Eve, PostGraphile, (also related frameworks that go the other other way, model to schema, e.g., Django REST), etc. (maybe overview of differences)
• Basic CRUD functions plus Auth
• Could we offload some backend functions of our apps to these frameworks (eg build on top of a CRUD API instead of the database directly)?
• They were primitive 5 years ago but some of them look pretty good now.
• Graphql (e.g., PostGraphile)

• Versioning
• Code review process and commenting (e.g., better way to review diffs than in the raw json notebook source? https://www.reviewnb.com/)
• Separating computation from presentation of results
• Debugging
• Latest developments with Jupyter Notebooks and JupyterHub

• Updates to design automatically flowing to dev
• Tools, now and coming
• Practices

• What are they (fonts, colors, padding/margin, etc.)
• How to get them right

• user database tables
• personal information

Use of SPARQL compliant stores

• Use case for SPARQL triplestores
• pros&cons of different stores
• additions: txt-search,geo,time,..
• property-graph bridges rdf*

• greater use of LinkedData in even simple FAIR metadata

• type hints
• data classes
• coroutines
• automatic code formatters (black)

• how to build a web service in graphql
• autogeneration of graphql endpoint based on database (???)

• effective and efficient use of Slack messaging
• parsing and storing important information
• what is expected, response time, priorities

• Basic concepts
• Fundamental algorithms
• Existing hardware architectures
• Currently accessible systems
• Open software frameworks
• Opportunities
• Potential for the IBM initiative on campus
• Focus group to build capacity and knowledge
• Look for experts (Eric Chitamber ECE)

• Language features
• The Mix build system/toolchain
• Unit testing
• Distributed processing with OTP
• Web development with Phoenix
• Hands on show and tell / Demo / overview
• Round table

• Pandas vs Numpy: when to use which?
• Four Pandas optimization strategies
• Numpy: harnessing vectorized operations

• Depends on the frontend tech
• Web browser
• Selenium
• Javascript Unit Testing - Mocha, etc.

Round Table

how to adapt agile practices when everyone on the team is assigned to multiple projects (and thus part of N simultaneous sprints)

possible complications:

• potentially incurring N times the meeting overhead for all the associated ceremonies
• deadline collisions
• costs of context switching
• potentially rebalancing time devoted to each project on a week-to-week basis according to needs thereby complicating any tracking of velocity

• in-application help in various forms, manuals, video tutorials, knowledge bases
• best practices for maintaining/updating/versioning these artifacts as part of the development process
• gathering requirements for documentation
• testing documentation
• tools and other resources
• how does this apply to scientists vs general users
  • docs for software engineers vs scientists
• Collecting information vs how to best write it
• Based on round table figure out focus

• Both working with Project Managers and basic Project Management skills developers can/should learn
• How to work closely with project managers on a bigger project 
• Tools such as JIRA/Confluence/GitHub Projects/Zenhub
• How to adapt agile practices when everyone on the team is assigned to multiple projects (and thus part of multiple simultaneous sprints)
• Dos and Don'ts

• why animate
• when and when not, what and what not
• css techniques
• js techniques
• libraries, guides, and other resources

Focus Group

TBD

• paradigm and mental model (callbacks, promises, observables)
• marble diagrams
• pros, cons, alternatives
• implementations (RxJS and the rest of the ReactiveX family (http://reactivex.io/languages.html), many others)
• use cases
• testing

• keycloak overview
• strengths and weaknesses
• hands on examples

• flask overview
• flask + connexion
• how to start a bare-minimum flask app
• hands on excercises

• overview of reactJs
• participants try to create an app together
• hooks
• redux examples

• workshops where participants deploy a services using docker/kubernetes on radiant together

Focus Group

TBD

• overall network structure and knowledge introduction
• How to put apache/nginx in front of an existing app
• request domain name and enable SSL at NCSA

• overview of popular components in AWS
• AWS has many tutorials which participants can try together:
  firehose + dynamodb + lambda + S3 etc
• Free educational accounts

• similar to above

Round Table First

Potential follow up focus groups to prepare materials

TBD

• a comparison between the three
• follow up to the specific ones

Round Table First

• react native
• flutter
• other
• how are people currently developing mobile apps for projects?
• Rokwire / Clowder / Farmdoc
• Tricks to make webapps easy to use on phones

Round Table

University policy on releasing an app

• releasing an app

Round Table

TBD

• mentioning SQL because we seem to be pretty familar with noSQL
• frequent used SQL commands
• maybe some quiz

• performance
• interoperability
• support
• using networkx, doesn't scale well
• common libraries and databaset

Focus Group

TBD

• I think we had a round table about this
• maybe more hands on?

• Web apps
  • HTML based analytics tools
  • Track hits, page load times and user engagement on site
  • google analytics, webvitals etc.
• Geographic location tracking
• Same for APIs - response times etc.
  • Tools that can used with nginx/apache or api gateways
• Standalone? As opposed to Google Analytics?

Hands-on Machine Learning Study Materials (Beginner to Intermediate Levels) for Research Software Engineers

• Come up with a set of good hands-on study materials that can be used by Research Software Engineers to develop and/improve machine learning skills.
• This needs to be useful for beginners as well as people having intermediate skills.

• Collate and adapt the collected materials if possible.

• If needed, the focus group may look at specific areas within the machine learning spectrum to reduce the scope.

Focus Group

TBD

Focus Group

TBD

• What makes a programming language functional?
• Scientific code and equational reasoning
• Immutability and correctness
• Higher order functions
• Algorithmic skeletons
• Lessons for code maintainability

Round Table

TBD

Focus group

TBD

Yes

• What is reproducibility?
• Conceptual models
• The effect of the software stack
• Architectural concerns
• Software design practices
• Can reproducibility become a design concern?

• Basic Introduction
• When should we use it
• How to scale
• Pros and Cons

• Highlighting (Oh-my-zsh, PowerLevel10k)
• Fuzzy search (fzf) – truly amazing.
• Auto python env activation (PyEnv and Auto-Conda)
• Delightful git integration (via fzf)
• AutoJump & Z (frequent dir access)

• jetbrains toolbox
• git clients
• git workflows
• pomodoro
• task tracking
• outlook viva insights
• Trello

• NCSA templates for dockerfiles, webapps,
  
  • boilerplate generator
• helm charts, docker compose

Tips and tricks:

• Easy-to-remember when to use different types of ML.
• My checklist when taking on an ML project (normalizing, feature engineering, checkpointing)
• Intro to modern ML ops tools like WandB.ai and ModelFlow (used on Radiant).

• Where to look for prospective candidates?
• What criteria use during the selection?
• How to maximze the likelihood of betting on the future "winner"?

Round Table

TBD