Transferring your Auth0 audit logs to Application Insights using a scheduled Webtask

If you're using the Microsoft Azure platform might be doing some monitoring of your applications with Application Insights. So wouldn't it be nice if you could take logs from third party services you're using (like Auth0) and also transfer these to Applications Insights? This would allow you to create a single dashboard in the Azure Portal in which you'll be able to see all of the errors, events, traces and metrics for all of your applications and services.

Auth0's Management API exposes an endpoint that allows you to retrieve the logs for your Auth0 account. So the only thing we need is a way to pull these logs and transfer them to Application Insights. And this should happen on a schedule (like once every five minutes).

There are plenty of ways you could do this, like a Worker Role in a Cloud Service. Wait what? Deploy a complete machine just for this, no thanks. Because in the end, what we're doing here is simple. Get some data from this API and send it to that API. So we could use a Web Job instead with the Azure Scheduler. Or maybe even something simpler?

What if I could just say run this code every 5 minutes. This is exactly what platforms like AWS Lambda or Webtask.io will allow you to do. The great thing about Webtask.io is that you don't have to deploy/upload anything. Just point to some code hosted anywhere (eg: GitHub) and schedule when you want to run it. Done.

Setting up your Webtask account#

We'll be using the Webtask CLI to schedule the task, so make sure you have Node.js installed and are running version 0.10.40 or higher.

To install the CLI and initialize your account just run:

npm i -g wt-cli
wt init

To verify if everything is setup correctly run wt profile ls and you'll see something like this:

Profile:    webtask
URL:        https://webtask.it.auth0.com
Container:  wt-sandrino-auth0_com-0
Token:      eyJhbGciOi...

Good.

Configuring Application Insights#

So next I'm going to create a new Application Insights in my Azure Subscription:

Wait a few second and once the provisioning is complete you'll be able to get the INSTRUMENTATION KEY from the Properties page:

Reviewing the code#

So the Webtask is fairly trivial:

• It gets the logs from Auth0
• It sends the logs to Application Insights
• It returns the last log (checkpointId), to know from where we need to continue next time (this is based on my colleague Soledad's Webtask that sends all logs to Loggly)

module.exports = (ctx, done) => {
  if (!ctx.data.AUTH0_DOMAIN || !ctx.data.AUTH0_GLOBAL_CLIENT_ID || !ctx.data.AUTH0_GLOBAL_CLIENT_SECRET) {
    return done({ message: 'Auth0 API v1 credentials or domain missing.' });
  }

  if (!ctx.data.APPINSIGHTS_INSTRUMENTATIONKEY) {
    return done({ message: 'Application Insights instrumentation key is missing.' });
  }

  /*
   * If this is a scheduled task, we'll get the last log checkpoint from the previous run and continue from there.
   */
  let checkpointId = null;
  if (ctx.body && ctx.body.results && ctx.body.results.length > 0) {
    console.log('Trying to get last checkpointId from previous run.');

    for (var i = 0; i < ctx.body.results.length; i++) {
      if (ctx.body.results[i].statusCode === 200){
        var result = JSON.parse(ctx.body.results[i].body);
        if (result && result.checkpointId) {
          checkpointId = result.checkpointId;
        }
        break;
      }
    };

    console.log('Starting from:', checkpointId);
  }

  const client = getClient(ctx.data.APPINSIGHTS_INSTRUMENTATIONKEY);
  client.commonProperties = {
    auth0_domain: ctx.data.AUTH0_DOMAIN
  };

  const auth0 = new Auth0({
    domain: ctx.data.AUTH0_DOMAIN,
    clientID: ctx.data.AUTH0_GLOBAL_CLIENT_ID,
    clientSecret: ctx.data.AUTH0_GLOBAL_CLIENT_SECRET
  });

  /*
   * Test authenticating with the Auth0 API.
   */
  const authenticate = (callback) => {
    auth0.getAccessToken(function (err, newToken) {
      console.log('Authenticating...');

      if (err) {
        console.log('Error authenticating', err);
        return callback(err);
      }

      console.log('Authentication success.');
      return callback();
    });
  };

  /*
   * Get the logs from Auth0.
   */
  const logs = [];
  const getLogs = (checkPoint, callback) => {
    auth0.getLogs({ take: 200, from: checkPoint }, (err, result) => {
      if (err) {
        return console.log('Error getting logs:', err.message);
      }

      if (result && result.length > 0) {
        result.forEach((log) => {
          // Application Insights does not allow you to send very old logs, so we'll only send the logs of the last 48 hours max.
          if (log.date && moment().diff(moment(log.date), 'hours') < 48) {
            logs.push(log);
          }
        });

        console.log(`Retrieved ${logs.length} logs from Auth0 after ${checkPoint}.`);
        setImmediate(() => {
          checkpointId = result[result.length - 1]._id;
          getLogs(result[result.length - 1]._id, callback);
        });
      }
      else {
        console.log(`Reached end of logs. Total: ${logs.length}.`);
        return callback(null, logs);
      }
    });
  };

  /*
   * Export the logs to Application Insights.
   */
  const exportLogs = (logs, callback) => {
    console.log('Exporting logs to Application Insights: ' + logs.length);

    logs.forEach(function(record) {
      var level = 0;
      record.type_code = record.type;
      if (logTypes[record.type]) {
        level = logTypes[record.type].level;
        record.type = logTypes[record.type].event;
      }

      ... Some mapping is happening here

      if (level >= 3) {
        var error = new Error(record.type);
        error.name = record.type;
        client.trackException(error, record);
      }

      client.trackEvent(record.type, record);
    });

    console.log('Flushing all data...');

    client.sendPendingData((response) => {
      return callback(null, JSON.parse(response));
    });
  };

  /*
   * Start the process.
   */
  authenticate((err) => {
    if (err) {
      return done(err);
    }

    getLogs(checkpointId, (err, logs) => {
      if (!logs) {
        return done(err);
      }

      exportLogs(logs, (err, response) => {
        if (response.itemsAccepted > 0) {
            console.log('Items accepted (Errors/Events):', response.itemsAccepted);
            return done(null, {
              checkpointId,
              ...response
            });
        }

        return done(null, response);
      });
    });
  });
};

The full code for this task is available on GitHub.

Scheduling the Webtask#

So now you'll just need to go to the Auth0 Management API Explorer where you'll find your Global Client Id (API Key) and Global Client Secret (API Secret).

With that we now have all of the information required to schedule our Webtask. Here's the command you'll run:

wt cron schedule \
    --name auth0-logs-to-appinsights \
    --secret AUTH0_DOMAIN="YOUR_AUTH0_DOMAIN" \
    --secret AUTH0_GLOBAL_CLIENT_ID="YOUR_AUTH0_GLOBAL_CLIENT_ID" \
    --secret AUTH0_GLOBAL_CLIENT_SECRET="YOUR_AUTH0_GLOBAL_CLIENT_SECRET" \
    --secret APPINSIGHTS_INSTRUMENTATIONKEY="YOUR_APPLICATION_INSIGHTS_INSTRUMENTATION_KEY" \
    --json \
    "*/5 * * * *" \
    https://raw.githubusercontent.com/sandrinodimattia/auth0-logs-to-appinsights-webtask/master/task.js

That's it. You just point to the script I have in my public GitHub repository, and it will run every 5 minutes with the secrets you specified.

No deployment, no build script, not even an npm install to install the Node.js modules we are using (Webtask comes with hundreds of Node.js modules included, which are all listed here).

And if you want to change the code, just fork my repository and point your Webtask to your fork.

The end result#

Ok great, your Webtask is scheduled to run every 2 minutes. Now what? Well what realtime logs? Just run wt logs and you'll see the realtime logs:

Afterwards, you can also manage the job:

• Remove the job using: wt cron rm auth0-logs-to-appinsights
• View the history of each run: wt cron history auth0-logs-to-appinsights

Finally when I got to Application Insights I can view every log with lots of details and nice overviews, allowing me to easily filter and search for logs.

I can now also use the alerting systems available in Application Insights to notify when something strange is happening (eg: send me an email when there's a lot of failed logins).

Or, what about adding this information to my dashboard, together with my other Azure resources?

What's next?#

Well, many things. Because this Webtask is also exposed through a url. So you can just call it from any HTTP enabled system. What does that mean? Webhooks. GitHub, Trello, Slack, ... can all just call your Webtasks and run your code. Again, without servers, without deploying, ...

Take a look here for a few samples that show how to integrate with these popular services: https://github.com/auth0/webtask-scripts

Have fun!