Antares Documentation

_images/antares.png _images/ANTARES-A.png

Welcome to Antares documentation.

API reference

The Arizona-NOAO Temporal Analysis and Response to Events System (ANTARES), at its core, analyzes camera alert data generated by the Large Synoptic Survey Telescope (LSST) in order to present astronomically interesting phenomena to scientists. Thus, the system determines and filters regular activity from the more interesting data astronomers wish to investigate further. In order to do this, ANTARES allows astronomers to develop code for use in determining the rarity of the observations.

When the LSST completes and transmits an exposure to ANTARES, that exposure is XOR’d with previous exposures of the same segment of the sky; differences between exposures are entered into ANTARES as a set of Camera Alerts containing the data from the exposure. Those Camera Alerts (and subsequently generated Alerts as will be discussed) are sent by ANTARES through the code developed by astronomers in stages which are run in a distributed system concurrently. These code stages may determine that a Camera Alert is not sufficiently interesting to continue analysis and will then divert the alert, which instructs ANTARES to not continue further processing; Camera Alerts not diverted will proceed to further code stages for further processing. In addition, stages may generate Alert Replicas- copies of a given Camera Alert associated with different Astro Objects (known astrometric phenomena)- for parallel stage processing (each Replica Alert may be processed separately) or for use in Combo Alerts- a type of Alert which allows multiple Alerts to be processed together in a given stage as opposed to individually as a single Camera Alert. Stages may also access External Alerts- data from other telescopes- as well as known Astro Objects- known astronomic phenomena- and use them in Alert Combos for processing.

Data for processing is located in the Alert object(s). Each Alert object organizes its data in Contexts and Properties. Alerts have Contexts, which in turn have Properties. Contexts allow for categorical organization of data. To illustrate, the Camera Alert context holds data related to the brightness of a Camera Alert and the Astro Object context holds currently known data (such as color intensity) regarding the Astro Object in question. These Contexts hold any number of Properties, which are variables generated by and/or used in the stage processing. Some Contexts have a set of predefined Properties populated by default, such as data inherently available in Camera Alerts from the LSST.

Within the ANTARES pipeline, stage code accesses the Alerts, Contexts, and Properties, as well as calling helper functions, through the ANTARES API. At the beginning of a stage, ANTARES- as defined in the API- passes the Alert for processing (including any Contexts and Properties associated with as well as Replicas in the case of a Combo Alert) to the stage code. ANTARES then retrieves this data from the Locus Aggregated Database (which is initially populated with data from the LSST). At the end of the stage, ANTARES stores data generated and used by stages in the Locus Aggregated Database for use in further stages. The helper functions supplied in the API, among other things, allows the stage code to direct what to do with an alert- discard, mark as rare, or simply continue processing- following the stage.

_images/api_processing.png

This figure demonstrates how ANTARES processes stage code. Here, workers are individual threads operating on nodes. They receive alert data to process from the master node and execute stages on that data as shown here.

This document specifies the ANTARES API, which allows development of stage code for execution on ANTARES.

Definitions

  1. An object may be “contained” by only a single other object. That other object may be considered the owner of the “contained” object.
  2. “Has” denotes a reference from one object to one or multiple others. Objects may be “had” by multiple other objects but are not owned by any of those objects.

Container Hierarchy

The way in which Contexts and Alerts are organized and linked directs how various pieces of data may be accessed and altered. The diagram below illustrates that organization.

_images/containerDiagram.png

Rules & Constraints

  1. Properties can be only accessed through the context that they belong to.

  2. Values of Properties are accessed directly by their names. For example:

    >>> value = alert.CA.GMinusR

    This will assign most-recently computed value for the property GMinusR to value. It will raise an AttributeError exception if CA context has no property called GMinusR.

  3. Properties are changed when we assign values to them. For example:

    >>> alert.CA.GMinusR = value

    This will assign value to the property GMinusR that is defined under CA context. It will raise an AttributeError exception if CA context has no property called GMinusR.

  4. Timestamp for a newly computed property value is assigned implicitly when the value is assigned to the property.

  5. Exceptions are handled by callers. The only exception in the API is Python’s built-in AttributeError exception.

  6. There will be a max size for strings. The max size is to be determined.

  7. The value of property may be NA (not applicable), but NK (not known) is not allowed as a value.

  8. The value for a Base Property is provided by the Camera in a camera alert or from the Aggregated AstroObject Catalogue for the relevant alert replica, and is always present when appropriate.

  9. The value for a Derived Property will be present for a derivation function for a given context if its arguments are present and if the algorithm for that derivation function completes.

  10. CA and LA Properties are always available.

  11. AR Properties are only accessible during per-replica processing; AO Properties are available if AR-HasAstroObject is True; ES only if AO-kind = "extended source"; PS only if AR-HasAstroObject is True and AO-kind = "point source".

  12. CB Properties are only visible during per-combo processing.

  13. CA processing can iterate through the alerts of its locus-aggregated alert to access AR, PS, and ES Properties. For example:

    >>> alert.LA.assembleTimeSeries_replics( 'AR', 'Redshift' )

    Here, alert is a camera alert and the above call returns a time series of all the values of the Property Redshift under AR context of the alert replicas associated with a locus aggregated alert.

  14. CB processing can iterate through its replicas to access AR, AO, PS, and ES Properties.

    >>> combo.CB.assembleVector( 'AR', 'Redshift' )

    Here, combo is an alert combo and the above call returns a numpy array of all the values of the Property Redshift under AR context of the alert replicas associated with the alert combo.

  15. A Derived Property cannot have a circular definition (and so definitions form a directed acyclic graph).

  16. Alert Replicas will not be processed until Camera Alerts have finished processing.

  17. Alert Combos will not be processed until Alert Replicas have finished processing.

Assumptions and Comments

  • Alerts are atomic; at any given point in time, only one stage will have access to a given alert and, redundancy concerns excepted, the alert will only exist in one state and in one place. Combos and replicas use duplicates of other alerts rather than references to the original alert.
  • The alerts (including Combo and Replica Alerts) can be processed entirely in parallel and entirely independently of one another. If a stage requires Properties from earlier stages, it will only be executed after those earlier stages (this in part relies on stage configuration). Only one stage may be executed on each Alert at a given time.

NOTE: we decided to leave for later discussion the identification of structures of camera alerts within an image (e.g., a cluster of say 100 alerts within a small area). We can incorporate such identification as a separate analysis, outside that of the stages. This may be done through the creation of Combo Alerts in stage processing, but algorithms independant of the stages may prove more effective.

  • Each Alert within a stage will have access only to Properties within that Alert.
  • Values of Property in a larger context (e.g., Image) will be the same across the relevant alerts (e.g., for every alert in that Image, for every alert in that RAFT). Larger contexts’ properties will derive from the aggregate of the smaller contexts.
  • Each alert within a stage will write a new value for one or more properties within multiple contexts.
  • ANTARES assumes that, once written at the end of a stage, a given property becomes final and immutable. Subsequent stages do not change the value of the property. This assumption may need to be reexamined as the stage code base grows beyond management.
  • Values cannot be written for a larger context (e.g., Image) within an alert stage. So a camera alert can write to properties in the CA context but not to the Image or AstroObject contexts. For more details on which contexts are writable, see Rules & Constraints above.

Note: the Antares Data Model document (https://docs.google.com/document/d/1xjYmhd8W9pyiwCBLA6o8mJeAz99Qbz9NvYFpICvYfSA/) had an example of

LA-LightCurve = F2( LA-CameraAlerts.TimeSeries(CA-Brightness), CA-Brightness, CA-Time )

Such an assignment would be allowed within a camera alert, because a Locus-Aggregated Alert (LA) has exactly one Camera Alert (CA) at any time. However, that would not be allowed within a replica or combo.

NOTE: We discussed computing something like the count of the number of alerts by allowing something like

IM-CountAlerts =+ 1

which reads IM-CountAlerts, adds one, and writes it. That is explicitly forbidden above. However, for such accumulative calculations, there may be ways to incorporate them by having a separate process that accepts such accumulations and coalesces them into a single assignment, rather than a complex sequence of parallel reads and write, which we want to avoid allowing.

  • Written values are stored in the database only when a stage completes. A stage that is aborted loses those values, because the stage was incomplete.
  • A property can be written to multiple times in a stage or in separate stages, but only the last value will be visible in subsequent stages or in output for a particular camera alert. (We allow this because different astronomers might have independently written different stages.)
  • The values form a time series, one value computed for each alert at that time. Hence, old values are retained and the new value extends the time series. No value is ever changed in the database (except for multiple writes, discussed above, which we can finesse.). In other words, each stage generates a new database entry. The entire history of alert processing will be present in the database, with one new write per stage and no modification of old (and perhaps stale) stage values.
  • There will be a separate conversion process that will translate the highly-optimized storage structure used in the ANTARES pipeline to and from the traditional relational structure as described in the ER diagram. The latter representation will be best for cross-alert analyses. The conversion(s) and cross-alert analyses will happen outside the time-critical portion of ANTARES and may be challenging in their own way in terms of performance.
  • User-contributed stages write to properties associated with a context of the current alert. So a single stage or multiple stages could successively write to the same property, changing its value as the processing pipeline progresses. We retain the value as of the end of each stage, but a future alert at the same locus will only see the final value of that property: there is no facility for seeing an intermediate value within a stage or between stages of a single past alert at that locus.
  • User-contributed stages can also write to local variables during their processing, but such variables will not be persisted between stages. The only values that are persisted are those associated with the contexts of the alert.

Indices and tables