Bayes’ theorem and rare disease

Bayes' Theorem is used to reverse the direction of conditioning. Suppose we want to ask what's the P(A|B) but we know it in terms of P(B|A). 

So we can write the P(A|B) = P(B|A) P(A) / P(B|A) P(A) + P(B| not A) P(not A)

This is same as P(A and B) / P(B)

This example is from an early test for HIV antibodies known as the ELISA test in North America.

Just for the example sake, I have replaced HIV with Covid19.

It's because this is a rare disease (see the probability of Covid19 in the screenshot ) and 

this is actually fairly common a problem for rare diseases. The number of false positives, 

greatly outnumbers the true positives because it's a rare disease. So even though the test is very accurate, we get more false positives than we get true positives. This obviously has important policy implications for things like mandatory testing. It makes much more sense to test in a sub population where the prevalence of Covid19 is higher, rather than in a general population where it's quite rare. 

Need to apply Deep Learning but don't have enough data, what to do next ?

Often it has been observed that analysts and data scientists want to apply deep learning models to solve the problem but they don't have enough data to train. There are three main ways to improve data: collecting more data, synthesizing new data, or augmenting existing data.. But what if not much academic work is there on the problem you want to solve. Convolutional Neural Networks have worked pretty well on most of the Computer Vision tasks. But all the CNN's (particularly deep CNN's) are heavily dependent on availability of very large training data to avoid overfitting. So in almost all computer vision tasks having more data help. In today’s world for the majority of Computer Vision tasks we don’t have enough data. So when you are training the computer vision model, often data augmentation is must.

Some of the common data augmentation used in Computer vision models are as given below.

a)    Mirroring

Below is the example of mirroring on the vertical axis.

b)    Random Cropping

It is not the ideal method for data augmentation but works well as long as your cropped images are reasonable subset of original image.

c)     Other Techniques like Rotation, Shearing and Local warping

d)    Color Shifting

Color shifting is about adding different distortion to RGB channels of an image.

Implementing distortions during training

We employ two distinct forms of data augmentation, both of which allow transformed images to be produced from the original images with very little computation, so the transformed images do not need to be stored on disk.

In our implementation, the transformed images are generated in Python code on the CPU while the GPU is training on the previous batch of images. So these data augmentation schemes are, in effect, computationally free.

A quick taxonomy of data augmentation method in general is depicted below for a big picture

References:

1)    Andrew NG Deep Learning deeplearning.ai

2)    Data Augmentation on Workera.ai

3)    AlexNet Paper on PCA color augmentation

4)    Datahackers.rs blog

Mistagging of information when you don't know your data

Finding out relevant articles related to an entity is an interesting task. It becomes complex when an entity is known with various acronyms and short forms. It becomes further complex when you have multiple entities with similar names, short names or acronyms.

The whole effort of complex web crawling and web scrapping framework using python scrappy, selenium etc. including tagging and presentation will go for a toss if articles and documents are not entity-tagged properly.

If you search South Indian Bank Stock on https:/moneycontrol.com/  website today (as on 30th Oct 19) and go to News & Research, the most recent and relevant articles you find for this stock is actually not related at all with South Indian Bank entity. Forget about title, you would not even find a mention of the entity, South Indian Bank anywhere inside the article. Actually it is related with an entity which is completely different but similar in name called Indian Bank.

Though money control website and mobile application are amazing in various aspects and it is one of the good sources of information for most of us who are active in share market but this kind of blunder does occur when you do not understand you data well.

Finding relevant articles related to an entity through matching has to be improved specially in these cases.

My Suggestion to moneycontrol Application-cum-AI architect would be to follow following simple steps while tagging.

Ø  Tag articles with entity name matches directly with Title text

Ø  Tag articles with entity name matches directly with Body text

Ø  Tag articles with entity name matches Partially but sufficient with Title text

o   Complex Fuzzy Match

o   Matches with Acronym

o   Matches with other short form

Ø  Tag articles with entity name matches Partially but sufficient with Body text

o   Complex Fuzzy Match

o   Matches with Acronym

o   Matches with other short form

Ø   Save the name of Matched and Matching entities along with Article IDs, steps etc.

Ø  Exclude an Article if it Matched entities directly matches with other matching entities.

AWS Solution Architect Associate Exam (Read Time - 4 Mins)

I passed my AWS Solution Architect Associate Exam couple of months back. Please find below useful tips on the same.

Before the exam:

• Go through the official AWS learning library: https://www.aws.training/LearningLibrary. It is entirely free & has the most updated information about AWS services.
• Complete the official AWS Exam Readiness: AWS Certified Solutions Architect (Associate) - Digital training (Free) :https://www.aws.training/learningobject/curriculum?id=20685
• Read the FAQ of each AWS Service. e.g., https://aws.amazon.com/vpc/faqs/
• Understand the AWS Well-Architected Framework & read each whitepaper from here: https://aws.amazon.com/architecture/well-architected/
• Take handwritten notes & make personalized cheat sheets whenever possible.
• Do plenty of hands-on practice. I had used Qwiklabs & it helped me a lot (https://www.qwiklabs.com)
• You need to understand how each AWS service can be tweaked for Cost, Quality, and Performance. How can you make S3 cheaper? How can you make it more redundant/secure? How can you make it more performant? What about DynamoDB or EBS? EC2? Etc.
• Take plenty of practice tests; it will give you confidence for the actual exam.

During the exam:

• Get plenty of rest before the exam day. It's very challenging to maintain concentration for 130 minutes, without any breaks.
• Read the answers first to understand what to focus on in the question.
• Read each question twice & make sure you have found the "keywords." It's the part of the question that tells you exactly what they want. e.g., "Which option provides the MOST COST EFFECTIVE solution."
• If you have no clue at first, eliminate wrong answers, then guess. Mark it for review and revisit it if you have time.

I hope you find this useful. And all the best for your exam !!

Apache Spark in Google Collaboratory

This is from my learning notes!!!

1.1    Setting up Spark on Google Colab

Google Collaborator is perfect cloud platform for someone to start learning Python. You can access what you practiced from anywhere and everywhere.

This could also be used to learn Spark . Please follow below steps. Make sure you check the file version and do the modification as needed (like look for latest .tgz file etc.)

1.1.1    Install Java, Spark, and Findspark

!apt-get install openjdk-8-jdk-headless -qq > /dev/null

!wget -q http://apache.osuosl.org/spark/spark-2.4.3/spark-2.4.3-bin-hadoop2.7.tgz

!tar xf spark-2.4.3-bin-hadoop2.7.tgz

!pip install -q findspark

1.1.2    Set Environment Variables

import os

os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"

os.environ["SPARK_HOME"] = "/content/spark-2.4.3-bin-hadoop2.7"

1.1.3    Start a SparkSession

import findspark

findspark.init()

from pyspark.sql import SparkSession

spark = SparkSession.builder.master("local[*]").getOrCreate()

1.1.4    Use Spark!

df = spark.createDataFrame([{"winner": "Humanity"} for x in range(100)])

df.show(2)

SAS Regular Expression Example

Below is the example of  SAS Regular Expression function to make you understand this.

Two Perl Regular Expression(PRX) Functions –

1.       PRXPARSE

Description - It define a Perl regular expression which is further used by other Perl Regular Expression function like PRXMATCH.

Syntax – PRXPARSE(“/Perl Regular Expression/i”)

                  “ ” à Part of SAS syntax

                  / à Default Perl delimiter

                I à Ignore case sensitive

Example à PRXPARSE(“/sas/i”)

2.       PRXMATCH

Description – To locate the position in a string, where a regular expression is matched. This function always returns the first position in a string expression of the pattern described by the regular expression.  If pattern is not found, then returns a zero.

Syntax – PRXPARSE(“/Perl Regular Expression/i” or Pattern_id, String)

                  “ ” à Part of SAS syntax

                  / à Default Perl delimiter

                I à Ignore case sensitive

                Pattern_id à is the value returned from the  PRXPARSE function

Example à PRXMATCH(“/sas/i”, String)

or

                       If _N_ = 1 then Pattern = PRXPARSE(“/sas/i”);

                       Retain Pattern;

                      PRXMATCH(Pattern, String)

Code –

To find the word “SAS” anywhere in the string.

DATA Test;   

IF _N_ = 1 THEN PATTERN_NUM = PRXPARSE("/sas/i");   

* match for the word 'SAS' anywhere in the string;   

RETAIN PATTERN_NUM;

INPUT STRING $30.;   

POSITION = PRXMATCH(PATTERN_NUM,STRING);   

FILE PRINT;   

PUT PATTERN_NUM= STRING= POSITION=;

DATALINES;

Welcome to SAS india

SAS with Perl regular expression

Enjoy SAS with PRX

Perl Regular expression

;

run;

Output-

                              

SAS Functions for File Operation : Basic Level

The Below code to understand SAS Functions related to Directory. You might not be using them in case you are using SAS metadata based tools but it is always advantageous to understand them .

This Code will provide the list of files and folders available within specific directory(List of Members within Directory).

Note: File could be with any extension(.sql, .sas, .txt, .xls & etc.)

Data Work.Test / view=work.Test;

/*Data _Null_;*/

Drop RC DID i;

RC = filename("Mydir", "G:\Test");

put RC;

did = dopen("Mydir");   /* Dopen  - open the directory and returns with directory identifier */

Put did;

if did > 0 then

      do i=0 to dnum(did); /* DNum  - returns number of members in a directory */

      dset = dread(did, i);   /* Dread  - returns the name of directory Member.  Dset will hold file name with extension and also folder name(if available) */

      dset1 = scan(dset,1,'.'); /* Dset1 will hold only file name also folder name(if available) */

      Ext = scan(dset,-1,'.'); /* Ext will hold only file extension also folder name(if available) */

      output;

end;

RC = dclose(did); /* DClose  - Close the Directory Opened by DOpen Function */

run;

Contributed by Shoaib Ansari