One topic from the previous meeting:
- the FrDF team is looking at developing a better
CSV translator (to be written in C++) with a possible option of integrating the one into the partitioning tools
- any news here?
Fabrice Jammes and Igor Gaponenko discussed an option for formal schema validation:
- the Ingest API's services will report the expected schema in some formally recognized schema definition format
- the ingest workflows would have to employ the corresponding formal schema validation tools to make sure the JSON files to be used by the workflow are compatible with the expected schema
- the very same should be done for the
qserv-ingest 's own file
metadata.json since the schema of the file may also evolve over time, and different copies of the files might be laying around (GitHub, file systems, etc.).
- altogether, this early checking should reinforce the ingest campaigns and prevent the "midflight" failures
Fabrice Jammes Where to put the documentation on the versioning of
- Fritz Mueller: unlike the Ingest API,
qserv-ingest is facing operations. So, it needs to be documented in the relevant (for operations) area of the documentation.
Igor Gaponenko: there is a new feature of the Ingest system that's about to be merged into the
main branch of Qserv:
DM-36722Getting issue details...