Bulk Data Upload

Purpose

There are several scenarios where you might have data collected in spreadsheets that need to be uploaded to Avni. For example

You are migrating away from an existing implementation, and need to seed with existing data.
Your organization has a separate component where data is collected outside Avni, but you still need this data to be present with field workers using Avni.

The two approaches

Avni has two mechanisms of uploading data -

~~the Microsoft Excel approach and~~
through the admin UI using CSV files

Uploading data using the Microsoft Excel approach is documented in Importing Excel data. This is more feature-rich, and as a result, has a steeper learning curve. This approach was also deprecated long time ago and is no longer supported.

Below, we discuss the UI approach to uploading data.

Using the Admin app to upload data

The Admin app of the web console has an upload option. Currently, this supports

Upload subjects
Upload program enrolment (excluding exit observations)
Upload program encounters (excluding cancel observations)
Upload encounters
Upload locations
Upload users and catchments
Upload metadata zip file downloaded from a different implementation

Sample files are available in the interface. Download the file, fill in values and then upload. The file is in a CSV format.

Form validations and rules

All the entries in CSV are validated before saving to the database. Suppose a field is marked mandatory in the form and value is not provided in the CSV then upload fails giving the error that the mandatory field cannot be empty.
All form element and form element group rules are run during CSV upload, so if there is a value for any form element which is hidden then that value is ignored. This behaves similarly to how data entry is done from the web or mobile app.
New visits get saved based on the visit schedule logic.
Decisions are saved along with the observations based on the decision rule logic.

Questions and Answers

What is the Id field in every file?

The Id is an identifier for the row you are uploading. This is important to ensure that if you upload the same file twice, we do not create duplicate records. For import, this usually should map to the id from your previous system. For updates, you can specify the value for the Id field as the id from your previous system or the uuid generated by Avni when creating the record. If you have two different individuals or encounters to be uploaded, please ensure they are uploaded using different ids. If not, they will be overwritten. The Id can be any string.

What if I have a comma in my observation?

Wrap your observation in quotes.

What if I need to upload an observation whose concept is not specified in the sample file?

It is possible you have a computed value that is not part of the form that needs to be uploaded. Just add the concept name in the header, and it will be added to the observations.

Is the order of values important?

No. Columns can be in any order.

What if I have a concept called "Id"? This will mean there are two headers in the same file with the same name.

Unfortunately, the upload process does not support this scenario. You can potentially change the name of the concept for a little while until the upload is complete, and then change it back (if you are doing an initial import, this makes sense). If not, try changing the name of the concept (we do an exact case-sensitive string match, so you can change the concept name to something like "ID", and it should work fine).

How to upload data for the grouped form elements?

Columns for the grouped form elements are labeled as "Parent|Child". One can fill in the values for all the child form elements and it'll get saved as grouped observation.

How do I upload images?

For images, use a url that the avni server can download. Ensure that
- The images are a direct download link (not a redirect to a page that uses javascript to download)
- The image urls end with the image type. eg: https://somedomain.com/images/abc.png