Add thousands of items in Kontent Headless CMS using Azure Functions

Akshay Sura - Partner

25 Feb 2021

Share on social media

In this blog post, we will be covering how to add a large number of items via Kontent Management APIs. As usual for tasks such as this, we love using Azure Functions.

Our first task is to get the data, in our case we are pulling it from an FTP server. We download the CSV files and process row by row to add Resellers. In your case, you might be pulling it from an API or a data store.

[FunctionName("ProcessBrandReseller")] 
public static async void ProcessBrandReseller( 
[QueueTrigger("process-reseller-file")] string filename, 
[Queue("add-reseller")] CloudQueue addResellerQueue, 
ILogger log) 
{
     log.LogInformation("Resellers.ProcessBrandReseller:"); 
     try
     {
          if (!string.IsNullOrEmpty(filename)) { var client = new FtpClient( Environment.GetEnvironmentVariable("ResellerFTPUrl"), Environment.GetEnvironmentVariable("ResellerFTPUser"), Environment.GetEnvironmentVariable("ResellerFTPPassword")); MemoryStream stream = new MemoryStream(); bool result = client.Download(stream, filename); if (result) { var brand = Path.GetFileNameWithoutExtension(filename); stream.Position = 0; StreamReader reader = new StreamReader(stream, System.Text.Encoding.UTF8, true); TextFieldParser csvReader = new TextFieldParser(stream); csvReader.CommentTokens = new string[] { "#" }; csvReader.SetDelimiters(new string[] { "," }); csvReader.HasFieldsEnclosedInQuotes = true; csvReader.TextFieldType = FieldType.Delimited; csvReader.TrimWhiteSpace = true; csvReader.ReadLine(); int delayInSeconds = 5; while (!csvReader.EndOfData) { string[] fields = csvReader.ReadFields(); HeadlessResellerModel headlessResellerModel = new HeadlessResellerModel(); headlessResellerModel.HydrateFromBrand(fields, brand); var cqm = new CloudQueueMessage(JsonConvert.SerializeObject(headlessResellerModel)); await addResellerQueue.AddMessageAsync(cqm, null, TimeSpan.FromSeconds(delayInSeconds), null, null); delayInSeconds += 1; } client.MoveFile(filename, "processed/" + filename, FtpRemoteExists.Overwrite); } log.LogInformation($"Resellers.ProcessBrandReseller: Successful "); } } catch (Exception ex) { log.LogError(ex, $"Resellers.ProcessBrandReseller.Error: {ex.Message}"); } }

In the above Azure Function, we are processing all resellers from the CSV files and adding the messages into another queue.

We now must process each message from the queue and Upsert (Update or Insert) the reseller.

[FunctionName("AddReseller")] public static async void AddReseller( [QueueTrigger("add-reseller")] HeadlessResellerModel headlessResellerModel, ILogger log) { string codeName = string.Empty; log.LogInformation("Resellers.AddReseller:"); try { if (headlessResellerModel != null && !string.IsNullOrEmpty(headlessResellerModel.Latitude) && !string.IsNullOrEmpty(headlessResellerModel.Longitude)) { ManagementOptions options = new ManagementOptions() { ApiKey = Environment.GetEnvironmentVariable("ResellerKontentApiKey"), ProjectId = Environment.GetEnvironmentVariable("ResellerKontentProjectId") }; ManagementClient client = new ManagementClient(options); codeName = string.Format("n{0}", headlessResellerModel.Dealer_Number).ToLower(); ContentItemUpsertModel item = new ContentItemUpsertModel { CodeName = codeName, Name = headlessResellerModel.Display_Name, Type = ContentTypeIdentifier.ByCodename("reseller") }; ContentItemModel responseItem = await client.UpsertContentItemByExternalIdAsync(headlessResellerModel.Dealer_Number, item); ContentItemIdentifier itemIdentifier = ContentItemIdentifier.ByExternalId(responseItem.ExternalId); LanguageIdentifier languageIdentifier = LanguageIdentifier.DEFAULT_LANGUAGE; ContentItemVariantIdentifier identifier = new ContentItemVariantIdentifier(itemIdentifier, languageIdentifier); ContentItemVariantModel<HeadlessResellerModel> responseVariant = await client.UpsertContentItemVariantAsync<HeadlessResellerModel>(identifier, headlessResellerModel); log.LogInformation($"Resellers.AddReseller: Successful "); } } catch (Exception ex) { log.LogError(ex, $"Resellers.AddReseller.Error: {codeName} {ex.Message}"); } }

A few things to point out. First, we add the base item using the ContentItemUpsertModel, this item has to contain an identifier. We are passing in the external id, codeName, the name, and the type of the item. We cannot pass in any other values for this item at this moment.

Second, we pass in the model with all the information as an ItemVariant. We create the variant identifier using the ExternalId (you could use other identifiers) and Language.

Since we are doing an upsert, the reseller is added if not already there and updated if they already exist.

You can get more information on the Management API's on Kontent: https://docs.kontent.ai/reference/management-api-v2

If you have any questions, please get in touch with me. @akshaysura13 on Twitter or on Slack.

Follow us on Twitter

Follow us on LinkedIn

Follow us on YouTube

Sign up to our newsletter

Tags:

Share on social media

Akshay Sura

Akshay is a nine-time Sitecore MVP and a two-time Kontent.ai. In addition to his work as a solution architect, Akshay is also one of the founders of SUGCON North America 2015, SUGCON India 2018 & 2019, Unofficial Sitecore Training, and Sitecore Slack.

Akshay founded and continues to run the Sitecore Hackathon. As one of the founding partners of Konabos Consulting, Akshay will continue to work with clients to lead projects and mentor their existing teams.

Subscribe to newsletter