This notebook deals with categorical data support which is now added to Daru. With this Daru can handle categorical data.

In [1]:

require 'daru'

Out[1]:

true

Initialization¶

Initialize a vector whose data is categorical by specifying type: :category

In [2]:

dv = Daru::Vector.new [:a, 1, :a, 1, :c], type: :category

Out[2]:

Daru::Vector(5)
0	a
1	1
2	a
3	1
4	c

In [3]:

dv.frequencies

Out[3]:

Daru::Vector(3)
a	2
1	2
c	1

You can initialize it with some predefined categories even though they do not exist using categories option.

In [4]:

dv = Daru::Vector.new [:a, 1, :a, 1, :c], type: :category, categories: [:a, :b, :c, 1]

Out[4]:

Daru::Vector(5)
0	a
1	1
2	a
3	1
4	c

categories option initalizes new categories and also specify the order in which they should occur. So now if you see the frequency table it would be ordered with the order you specified.

In [5]:

dv.frequencies

Out[5]:

Daru::Vector(4)
a	2
b	0
c	1
1	2

Since categorical data can be ordered as well as unordered you can specify whether the vector is ordered or not using the ordered: true or ordered: false during initialization.

In [6]:

dv = Daru::Vector.new [:a, 1, :a, 1, :c], categories: [:a, :b, :c, 1], ordered: false, type: :category

Out[6]:

Daru::Vector(5)
0	a
1	1
2	a
3	1
4	c

In [7]:

dv.min

ArgumentError: Can not apply min when vector is unordered. To make the categorical data ordered, use #ordered = true
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/daru-0.1.3.1/lib/daru/category.rb:383:in `assert_ordered'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/daru-0.1.3.1/lib/daru/category.rb:216:in `min'
(pry):7:in `<main>'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:355:in `eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:355:in `evaluate_ruby'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:323:in `handle_line'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:243:in `block (2 levels) in eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:242:in `catch'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:242:in `block in eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:241:in `catch'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:241:in `eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/backend.rb:65:in `eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/backend.rb:12:in `eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/kernel.rb:87:in `execute_request'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/kernel.rb:47:in `dispatch'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/kernel.rb:37:in `run'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/command.rb:70:in `run_kernel'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/command.rb:34:in `run'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/bin/iruby:5:in `<top (required)>'
/home/ubuntu/.rvm/gems/ruby-2.2.1/bin/iruby:23:in `load'
/home/ubuntu/.rvm/gems/ruby-2.2.1/bin/iruby:23:in `<main>'
/home/ubuntu/.rvm/gems/ruby-2.2.3/bin/ruby_executable_hooks:15:in `eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/bin/ruby_executable_hooks:15:in `<main>'

As you can see you can't do the comparision if vector is not ordered. Lets make it ordered.

In [8]:

dv = Daru::Vector.new [:a, 1, :a, 1, :c], ordered: true, categories: [:a, :b, :c, 1], type: :category

Out[8]:

Daru::Vector(5)
0	a
1	1
2	a
3	1
4	c

In [9]:

dv.min

Out[9]:

:a

In [10]:

dv.sort!

Out[10]:

Daru::Vector(5)
0	a
2	a
4	c
1	1
3	1

#categories= and #categories¶

Beside during the initialization you can also set the categories after the vector has been initialized.

In [11]:

dv = Daru::Vector.new [:a, 1, :c, 1, :c], type: :category
dv.categories = [:a, :b, :c, 1]

Out[11]:

[:a, :b, :c, 1]

You can also check all the categories associated with the vector.

In [12]:

dv.categories

Out[12]:

[:a, :b, :c, 1]

#ordered= and #ordered?¶

You can specify if the vector has to be treated as ordered or not after initialization of vector.

Note: By default the vector will be unordered

In [13]:

dv = Daru::Vector.new [:a, 1, :c, 1, :c], type: :category
dv.ordered?

Out[13]:

false

In [14]:

dv.ordered = true
dv.ordered?

Out[14]:

true

#summary¶

Here are a few measures to summarize categorical vector.

In [15]:

dv = Daru::Vector.new [:a, :a, :a, :b, :b, :c], type: :category
dv.summary

Out[15]:

Daru::Vector(6)
size	6
categories	3
max_freq	3
max_category	a
min_freq	1
min_category	c

#frequencies¶

Gives the frequency of each category in the order they occur.

In [16]:

dv = Daru::Vector.new ['third']*3 + ['second']*2 + ['first'], type: :category, categories: ['first', 'second', 'third']
dv.frequencies

Out[16]:

Daru::Vector(3)
first	1
second	2
third	3

#min, #max and #sort!¶

Note: These operations only apply if the vector is ordered.

In [17]:

dv

Out[17]:

Daru::Vector(6)
0	third
1	third
2	third
3	second
4	second
5	first

In [18]:

dv.min

ArgumentError: Can not apply min when vector is unordered. To make the categorical data ordered, use #ordered = true
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/daru-0.1.3.1/lib/daru/category.rb:383:in `assert_ordered'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/daru-0.1.3.1/lib/daru/category.rb:216:in `min'
(pry):23:in `<main>'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:355:in `eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:355:in `evaluate_ruby'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:323:in `handle_line'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:243:in `block (2 levels) in eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:242:in `catch'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:242:in `block in eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:241:in `catch'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:241:in `eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/backend.rb:65:in `eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/backend.rb:12:in `eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/kernel.rb:87:in `execute_request'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/kernel.rb:47:in `dispatch'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/kernel.rb:37:in `run'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/command.rb:70:in `run_kernel'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/command.rb:34:in `run'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/bin/iruby:5:in `<top (required)>'
/home/ubuntu/.rvm/gems/ruby-2.2.1/bin/iruby:23:in `load'
/home/ubuntu/.rvm/gems/ruby-2.2.1/bin/iruby:23:in `<main>'
/home/ubuntu/.rvm/gems/ruby-2.2.3/bin/ruby_executable_hooks:15:in `eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/bin/ruby_executable_hooks:15:in `<main>'

In [19]:

dv.ordered = true

Out[19]:

true

In [20]:

dv.min

Out[20]:

"first"

In [21]:

dv.max

Out[21]:

"third"

In [22]:

dv.sort!

Out[22]:

Daru::Vector(6)
5	first
3	second
4	second
0	third
1	third
2	third

#add_category¶

Associates new categories with the vector.

Note: In order to insert a new categorical value you need to use #add_category to make sure this category is registered in the vector. For example -

In [23]:

dv

Out[23]:

Daru::Vector(6)
5	first
3	second
4	second
0	third
1	third
2	third

In [24]:

dv[0] = 'fourth'

ArgumentError: Invalid category fourth, to add a new category use #add_category
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/daru-0.1.3.1/lib/daru/category.rb:505:in `modify_category_at'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/daru-0.1.3.1/lib/daru/category.rb:144:in `[]='
(pry):29:in `<main>'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:355:in `eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:355:in `evaluate_ruby'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:323:in `handle_line'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:243:in `block (2 levels) in eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:242:in `catch'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:242:in `block in eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:241:in `catch'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/pry_instance.rb:241:in `eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/backend.rb:65:in `eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/backend.rb:12:in `eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/kernel.rb:87:in `execute_request'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/kernel.rb:47:in `dispatch'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/kernel.rb:37:in `run'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/command.rb:70:in `run_kernel'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/lib/iruby/command.rb:34:in `run'
/home/ubuntu/.rvm/gems/ruby-2.2.3/gems/iruby-0.2.9/bin/iruby:5:in `<top (required)>'
/home/ubuntu/.rvm/gems/ruby-2.2.1/bin/iruby:23:in `load'
/home/ubuntu/.rvm/gems/ruby-2.2.1/bin/iruby:23:in `<main>'
/home/ubuntu/.rvm/gems/ruby-2.2.3/bin/ruby_executable_hooks:15:in `eval'
/home/ubuntu/.rvm/gems/ruby-2.2.3/bin/ruby_executable_hooks:15:in `<main>'

In [25]:

dv.add_category 'fourth'
dv[0] = 'fourth'
dv

Out[25]:

Daru::Vector(6)
5	first
3	second
4	second
0	fourth
1	third
2	third

In [26]:

dv.categories

Out[26]:

["first", "second", "third", "fourth"]

#rename_categories¶

You can rename subset of existing categories by passing a hash mapping old ones to new ones.

In [27]:

dv = Daru::Vector.new [1, 2, 'third', 2, 1], type: :category

Out[27]:

Daru::Vector(5)
0	1
1	2
2	third
3	2
4	1

In [28]:

dv.rename_categories 1 => 'first', 2 => 'second'
dv

Out[28]:

Daru::Vector(5)
0	first
1	second
2	third
3	second
4	first

Indexing¶

#[], #[]=, #at, #at_set¶

Indexing works similar to an ordinary vector, so you can expect these methods to do the same as with ordinary vector. Here are few examples:

In [29]:

dv = Daru::Vector.new [1, 1, 2, 2, 3, 1], index: :a..:f, type: :category

Out[29]:

Daru::Vector(6)
a	1
b	1
c	2
d	2
e	3
f	1

In [30]:

dv[0..2]

Out[30]:

Daru::Vector(3)
a	1
b	1
c	2

In [31]:

dv.at -1

Out[31]:

In [32]:

dv.set_at [0, 1], 3
dv

Out[32]:

Daru::Vector(6)
a	3
b	3
c	2
d	2
e	3
f	1

Querying¶

#where¶

Daru uses Arel-like syntax for querying data.

In [33]:

dv = Daru::Vector.new ['I', 'II', 'I', 'III', 'IV', 'I', 'II'], type: :category, categories: ['I', 'II', 'III', 'IV']
dv.ordered = true
dv.frequencies

Out[33]:

Daru::Vector(4)
I	3
II	2
III	1
IV	1

In [34]:

dv.where(dv.eq('I'))

Out[34]:

Daru::Vector(3)
0	I
2	I
5	I

In [35]:

dv.where(dv.gt('II'))

Out[35]:

Daru::Vector(2)
3	III
4	IV

In [40]:

df = Daru::DataFrame.new({
  a: (1..7).to_a,
  b: ('a'..'g').to_a,
  c: ['I', 'II', 'I', 'III', 'IV', 'I', 'II']
  })

Out[40]:

Daru::DataFrame(7x3)
	a	b	c
0	1	a	I
1	2	b	II
2	3	c	I
3	4	d	III
4	5	e	IV
5	6	f	I
6	7	g	II

In [41]:

df.c = df.c.to_category
df

Out[41]:

Daru::DataFrame(7x3)
	a	b	c
0	1	a	I
1	2	b	II
2	3	c	I
3	4	d	III
4	5	e	IV
5	6	f	I
6	7	g	II

In [46]:

df.where(df.c.gt('I') & df.c.lt('IV'))

Out[46]:

Daru::DataFrame(3x3)
	a	b	c
1	2	b	II
3	4	d	III
6	7	g	II

Contrast coding¶

Categorical data supports 4 types of contrast coding schemes-

Dummy Coding (Default)
Simple Coding
Helmert Coding
Deviation Coding

In [50]:

dv = Daru::Vector.new ['I', 'II', 'I', 'III', 'IV', 'I', 'II'], type: :category, categories: ['I', 'II', 'III', 'IV']
dv.name = 'Rank'
dv.contrast_code

Out[50]:

Daru::DataFrame(7x3)
	Rank_II	Rank_III	Rank_IV
0	0	0	0
1	1	0	0
2	0	0	0
3	0	1	0
4	0	0	1
5	0	0	0
6	1	0	0

You can set the base category using #base_category=

In [51]:

dv.base_category = 'IV'
dv.contrast_code

Out[51]:

Daru::DataFrame(7x3)
	Rank_I	Rank_II	Rank_III
0	1	0	0
1	0	1	0
2	1	0	0
3	0	0	1
4	0	0	0
5	1	0	0
6	0	1	0

To use any other coding using #coding_scheme

In [55]:

dv.coding_scheme = :deviation
dv.contrast_code

Out[55]:

Daru::DataFrame(7x3)
	Rank_I	Rank_II	Rank_III
0	1	0	0
1	0	1	0
2	1	0	0
3	0	0	1
4	-1	-1	-1
5	1	0	0
6	0	1	0

In [ ]: