| SLAC / Data Access - Qserv | Igor Gaponenko | - Stripe 82 catalog databases have been loaded into Qserv; simple tests seem to be working
- Have not tried performance / large-scale tests yet
- Lots of changes to the loading procedure were required in order to make this efficient & correct
- Procedure is now highly parallel
- Next step: produce a writeup on loading, especially for Unknown User (speckins)'s use
- Should be possible to experiment with loading additional databases without affecting S82 data
- Working on better file organization for the image data
- Looking at following Unknown User (jalt)'s proposal for data immutability : xxx(see
Jira |
---|
server | JIRA |
---|
columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution |
---|
serverId | 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 |
---|
key | RFC-249 |
---|
| )
- Ran into some problems with respect to time stamps / time zones in creating Docker containers (see
Jira |
---|
server | JIRA |
---|
columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution |
---|
serverId | 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 |
---|
key | DM-8119 |
---|
| )- Interpreting logs when containers have a different time zone than the host OS - KTL says that
Jira |
---|
server | JIRA |
---|
columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution |
---|
serverId | 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 |
---|
key | DM-1203 |
---|
| - Set single time for all project software & common machines Done is meant to be the current policy on log timezones and formatting; if NCSA can now support UTC-only systems, that would be preferable. - Interpretation of time stamps in the database data - KTL says that database columns are meant to be explicit in the schema (usually in the column name itself) as to which time scale they are in (TAI or UTC) and should never be in local time. But while this appears to have been followed in the baseline schema, it doesn't seem to have been written down. MySQL (MariaDB) appears to store TIMESTAMP columns as integers containing YYYYMMDDHHMMSS interpreted as UTC but they appear to ignore leap seconds and do not accept 60 as a seconds value.
- Unknown User (jalt): NCSA is very happy to go to uniform UTC time zone settings in production services
|