Jim asked about what kind of API is available to HTCondor. Greg T. said there is a Python API. This info is in the HTCondor manual. Jim will investigate
Greg D. asked about other file transfer mechanisms. Greg T. said there are file transfer plugins available on the client side that can access data in a variety of ways, using URLs. The "gotcha" part of this is that the plugins have to be smart about how to deal with problems (intermittent and otherwise), so they do their own retries to be more robust.
Greg D. discussed a cpu affinity issue that he saw when using the application launcher on Blue Waters which manifested itself as the HTCondor jobs not using as much of the CPU resources as expected. When using the cluster compatibility mode of Blue Waters, HTCondor ran the jobs fully using the CPUs, which was expected. We kicked around a few different ideas of how to go about debugging this, and will consult with the help desk at NCSA to see what their tools might be able to tell us.
As an aside, Greg D. said that DES is starting to use HTCondor on Illinois' Campus Cluster system.