1,041 different number-one hits between Feb 2015 and Aug 1958 seems too few. 

1041/57=19 songs a year or, spun another way, is like singing the same song for 2.5 weeks... On second thought - that sounds about right.

library(XML)

theurl <- "http://en.wikipedia.org/wiki/Billboard_year-end_top_100_singles_of_1958"

tables <- readHTMLTable(theurl)

write.csv(tables[[1]] [2:3], file = "Billboard Hot100 1958.csv", quote=FALSE)

baseurl<-"http://en.wikipedia.org/wiki/Billboard_Year-End_Hot_100_singles_of_"

for (yr in 1959:2014 ) {

#Wikipedia markups such as "This article has multiple issues" will break this loop

    stringyr <- as.character(yr)

    theurl <- paste0(baseurl, stringyr)

    tables <- readHTMLTable(theurl)

    fileName <- paste0("Billboard Hot100 ", stringyr, ".csv")

    write.csv(tables[[1]] [2:3], file = fileName, quote=FALSE)

}

Low: 3:00*1041=3123 minutes -> 52 hours

High: 4.50*1041=4685 minutes -> 78 hours

write.csv(x, file = "foo.csv")

read.csv("foo.csv", row.names = 1)

read.csv(file_name, header = TRUE, sep = ",")

write.csv(tables[[1]] [2:3], file = "foo.csv", quote=FALSE)

1) According to assumption "if it's Billboard ranked... it's popular enough" instrumental song counts should be reported but perhaps linked to a separate chart. 

2) ...

result = tryCatch({

    expr

}, warning = function(w) {

    warning-handler-code

}, error = function(e) {

    error-handler-code

}, finally = {

    cleanup-code

}

Scatterplot3d –an R package for Visualizing Multivariate Data

3D Plotting - Paul E. Johnson's

Apparently he, RhettAllain, writes for Wired Science. 

I like his error bars.

n = c(2, 3, 5) 

s = c("aa", "bb", "cc") 

b = c(TRUE, FALSE, TRUE) 

df = data.frame(n, s, b)       # df is a data frame 

library(stringr)

library(plyr)

del <- colwise(function(x) str_replace_all(x, '\"', ""))

x <- del(x)

URL <- "http://www.ipchicken.com/"

doc = htmlTreeParse(URL)

automating the login to the uk data service website in R with RCurl or httr

Use RCurl to bypass “disclaimer page” then do the web scrapping

Apostrophes are turned into a single dash: That's=That-s

Spaces are turned into a single dash: Hello There=Hello-There

Commas are removed

Parenthesis are removed

No consecutive dashes

No ampersands

Periods are turned into a single dash (this breaks the URL if it produces a dash as last character): T.I.=T-I-

No plus signs

No exclamation marks

"featuring" is turned into "feat"

Dollar signs are turned into single dash: Ke$ha=Ke-ha

No forward slashes (/)

It should look like www.blaI'mNotSinglingOutSourcebla.com/lyrics/Artist-name-separated-by-dashes/Song-name-separated-by-dashes

Throw flag for every artist/song combo that misses landing.

Condense flagged songs into error list that can be triaged either programmatically or manually.

For option 2:

There is a greater chance of scraping the wrong song with this option. Easy remediation dubious.

install.packages('data.table')

library(data.table)  #quickest ways to learn the features is to type example(data.table) and study the output.

fileName <- paste0("Billboard Hot100 ", year, ".csv")

nextFile<-read.csv(fileName, header = TRUE, sep = ",")

write.table(nextFile,

            "Master Billboard.csv",

            sep = ",",

            dec = ".",

            qmethod = "double",

            row.names = FALSE,

            append = TRUE,

            col.names = FALSE)

Hmmmm, interesting. I checked to see who created that article and unfortunately he has not been active since 2014. I do know that in the past Billboard did have songs tied, both in weekly or yearly charts, although I didn't realize it happened as recently as '69. Both billboard.com and billboard.biz have year-end rankings going back to only 2002. I'll have to research around to see what I find. - eo (talk) 13:17, 6 March 2015 (UTC) 

myfunction <- function(arg1){

    print(arg1)

    print(arg1*2)

}

myfunction(3)

Use stringsAsFactors=FALSE

Check out my total newb question on SO - Subsetting by [ ] returns a plurality of elements when only a single element was expected [R]. Why?

a<-"dlkflk-"

vec <- c("a", "b", "c","c","c")

vec[duplicated(vec) | duplicated(vec, fromLast=TRUE)] ## [1] "c" "c" "c"

master<-read.csv("Master Billboard.csv", stringsAsFactors=FALSE)
titleArtist<-paste(master$Title,master$Artist)
duplicatesIndex<-which(duplicated(titleArtist) | duplicated(titleArtist, fromLast = TRUE))
duplicateList<-master[duplicatesIndex,]

-Determine initial release dates of songs and flag any entries appearing earlier in time.

-Work with Wikipedia moderators to track integrity of rankings.

-Manually scour the Internet for correct data on dubious entries - ain't nobody got time for that.

if(condition == TRUE) {x <- TRUE

else x <- FALSE}

 if (type == "mean") 1
 else if (type == "median") 2
 else if (type == "trimmed") 3

...or switch()

UPDATE 2:

It appears that what is crossed out above also is not a true switching statement in the sense that if the first statement is executed, the second two are not. That's bogus.

getURL(u,.opts=curlOptions(followlocation=TRUE))

getURL(url='https://www.bla.cm', curl=my.handle, .opts = list(ssl.verifypeer = FALSE))

getURL(url='https://www.bla.cm', curl=my.handle, .opts = list(ssl.verifypeer = FALSE,followlocation=TRUE))

chars <- "test"

value <- "es"

grepl(value, chars)

                  <!DOCTYPE html><html><head>
  <title>Page 2 :)</title></head><body>
  This is a page 2.
  <a href="page1.html" title="to page 1">Want to go back to page 1? Click here</a></body></html>
                

<h1> - <h6>	Heading	<h2>Headings Are Great Fun</h2>
<p>	Paragraph	<p>This is my first. I hope you like it.</p>
<i>	Italic	<b>italicized word</b>
<b>	Bold	<b>bold word</b>
<a>	Anchor (hyperlink)	<a href="http://www.google.com">Link to Google</a>
<ul> & <li>	Unordered List & List Item	<ul> <li>Apples</li> <li>Bananas</li> <li>Pears</li> </ul>
<blockquote>	Blockquote	<blockquote>“To be or not to be, that is the question.” - Person who said this</blockquote>
<hr>	Horizontal Rule (draw line)	<hr />
<img>	Image	<img src="myimage.jpg" />
<div>	Division	<div>   other tags here  </div>      ...Used for dividing web page content into containers

FALSE    TRUE    NA's
182        5315     5 

3.3%      96.6%  0.1% 

If less than 5% missing, spend 2 hours getting remainder. Worry about any others missing after core analysis is developed.

Else, spend 4 hours getting as many lyrics as you can or until less than 5% missing.

Hypothesis 1

Lyrically, the most popular songs in America have become less unique (more repetitive) over time.

Hypothesis 2

Not really a hypothesis but it would be cool to analyze song themes and plot distributions. Possible themes include;

Love - Deeply in Love

Love - Breakups

Lust

Drugs

Partying

Travel

A million other things

Hypothesis 3

 The most common lyrical noun or verb of the most popular songs in America is "love"

The average popular song in America is written at lower than an eighth grade reading level (as measured by readability indices such as the Gunning Fog). 

Flesch–Kincaid Reading Ease:

Score	Notes
90.0–100.0	easily understood by an average 11-year-old student
60.0–70.0	easily understood by 13- to 15-year-old students
0.0–30.0	best understood by university graduates

Flesch–Kincaid Grade Level:

The number of years of education generally required to understand a text relevant when the formula results in a number greater than 10. 

d<-d[!(d$A=="B" & d$E==0),]

Duplicates                                  Regular                                    Instrumental

Duplicate:198

Regular:5430

Instrumental:71

Incomprehensibly Foreign: 1 (PSY's Gangnam Style)

198+5430+71+1=5700

1. HOME - Motivation
2. 'XXX' Questions (from this journal)
3. Analysis (Results)
4. Source Code - Link to Git
5. An About Me page

 DUPLICATE , TITLENAME, ARTISTNAME

for %f in (*.txt) do type "%f" >> output.txt

Miscellaneous

 Read more:

x <- "I like %$@to*&, chew;: gum, but don't like|}{[] bubble@#^)( gum!?"

gsub("[^[:alnum:][:space:]']", "", x)

[1] "I like to chew gum but don't like bubble gum"

http://ftp.gnu.org/gnu/aspell/w32/Aspell-0-50-3-3-Setup.exe

C:\Users\UrPC\AppData\Roaming\SpellCheckers\Aspell\bin\

http://ftp.gnu.org/gnu/aspell/w32/Aspell-en-0.50-2-3.exe 

Step 6. Run the Aspell dictionary 'installer' which will unpack itself to a folder called TmpInstall in the same directory.

Step 7. Open the TmpInstall folder and run setup-Aspell-en-0.50-2.exe. It should auto detect where your Aspell Win32 installation is based on the system path we added in Step 3. If not, proceed to Step 8.

Step 8. Open windows PowerShell and type the word

aspell

You should see command information vomit down your prompt. If you don't you may have entered the PATH information incorrectly (and need to fix it) or need to restart Windows. If after doing this you still throw an error, consider the Appendix.

Appendix: If Aspell was previously uninstalled you may need to delete it's orphaned registry key. Run regedit.exe and search for aspell. Delete any aspell key associated with an uninstalled aspell directory. Be very careful not to unintentionally change anything other than the aspell key in question as deleting something by accident could send you to Windows Hell.

 

src<-"http://svn.R-project.org/R/trunk/src/library/stats/man/lm.Rd"

f<-"lm.Rd"

download.file(src, f)

a <- aspell(f, "Rd")

print(subset(a,Original %in% c("ANOVA", "regressor")),verbose=TRUE)

1. Further minimize the file to be corrected by removing words that appear twice due to capitalizations, i.e. Don't need

Hello, hello, heLLo

to be counted three times. Ignore caps!

2. Run aspell() on output of 1.

3. Train aspell(), that is create a dictionary, that it will use to auto-correct misspelled words in the future.

... OR...

create a hash table that maps only the mispelled words to

4. Use the new dictionary to systematically correct each of the 5500+ unique lyric files (save to new file)

5. Run unique word count analysis on each of the corrected files from 4.

6. Plot and summarize results

Solution 1. Inspect and tag all files that have parenthesis. Then manually (or if better process evident) remove extraneous information.

Solution 2. Use gsub() to replace all instances of certain Unicode characters.

1. If word is known, mark Known=TRUE

    else, mark Known=False

Sub-algo 1a. If difference between suggested respelling and word is case, accept suggestion.

Sub-algo 2a. Identify all words with apostrophes after word.

Sub-algo 2b. Substitute the letter g for apostrophe

Sub-algo 3a. Identify all words with apostrophes before word.

Sub-algo 3b. remove apostrophe

Sub-algo 4a. Identify all words that are strung togetherlikethis

Sub-algo 4b. Separate such words using the SplitWords() function

Sub-algo 5a. Identify all words with repeated letters. Perhaps ignore some repetitions such as 'pp'

Sub-algo 5b. Eliminate all repeated letters.

2. Validate output of Sub-algos with dictionary cross check.

if reduced word is a known word, consider said word correct

else, seperate error to bin for further analysis.

3. Repeat 1-2 using a different sub-algo each pass until no word is marked unkown. 

4. If word is found to be spelled correctly but is not in dictionary, add said word to main dictionary.

Note that spell checking a corpus thousands of words long in this way will always retain residual error and that further reduction of this error requires significant expense (time/money). An automated abridged spell check of this nature is therefore useful considering it's noise reduction per effort ratio. 

Don't loose sight of the fact that The ultimate desire is to obtain a function that corrects misspelled words in place.

aspell("test bag of words.txt", control = c("--master=en_US","-extra-dicts=es"))

aspell("test bag of words.txt", control = c("--master=en_US","-extra-dicts=es"))

aspell() Windows Binary - A reply to your 10 year old invitation

Hello Dr. Temple Lang

We've never met and probably never will. I, however, have had the benefit of being able to use your documentation of aspell() and other help files to work my way through several coding uncertainties - thank you! I write to you now wondering specifically about your package aspell()'s functionality on windows machines and whether procuring a windows binary of the package at install.packages("Aspell", repos = "http://www.omegahat.org/R"), as you mentioned you may supply if one were to request it from you, was possible. I realize it's been a while since various portions of your aspell() documentation have been updated and understand if new life directions have led to it's abandonment. I have been considering using Python to handle text preprocessing needs and, if you cannot provide a windows binary of your aspell wrapper, would you be able suggest an alternative?

Have a good one,

Andrew Agostini

install.packages("deducorrect")

library(deducorrect)

1969     90     Instrumental

1977     21     Instrumental

1980     43     Instrumental

2012     47     Gangnam Style     PSY

Lexp[!(Lobs %in% Lexp)]

Lexp[is.na(match(Lexp,Lobs))]

              
                cat ("Press [enter] to continue")
              
            

write("\r",file="data.txt", append=TRUE)

1. A song void of words spoken in some language

2. A song which is obviously popular because of it's instrumental score AND NOT the few words it may contain. Only a few songs fall in this category

English

Spanish

Similar Latin languages such as Italian and written French (context always helps)

I once had a ten minute conversation with an Italian couple in the Grand Canyon where I spoke Spanish, they spoke their native tongue, and the three of us understood each other impressively well. # A toast to awesome connections!

Dime San Antonio

Dime donde vas

Parado de manos

¿Dónde llegarás?

(llegarás, llegarás)

Dime San Antonio

Dime donde vas

Parado de manos

¿Dónde llegarás?

(llegarás, llegarás)

                Authenticate
              

                Plot something
              

library(plotly)

dsamp <- diamonds[sample(nrow(diamonds), 1000), ]

qplot(carat, price, data=dsamp, colour=clarity)

py <- plotly()

py$ggplotly()

                Make graph and publish to JSON or JavaScript. 
                Alternatively one could embbed the code
              

                UPDATE 1:
              

                Although it is awesome and powerful, I will not be using Plot.ly
              

Scripts can also be placed in external files.

External scripts are practical when the same code is used in many different web pages.

JavaScript files have the file extension .js.

To use an external script, put the name of the script file in the src (source) attribute of the <script> tag:

Example

<!DOCTYPE html>

<html>

<body>

<script src="myScript.js"></script>

</body>

</html>

> dmaster[4502,]

     Year Rank     Title        Artist Duplicate Instrumental LineCount uCount

4666 2004   66 One Thing Finger Eleven     FALSE        FALSE        53      3

> length(unique(scan("2004 66.txt", character(0))))

Read 3 items

[1] 3

Warning message:

In scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings,  :

  EOF within quoted string

Set DNS to

ns1.asoshared.com

ns2.asoshared.com

• S – Single-responsibility principle
• O – Open-closed principle
• L – Liskov substitution principle
• I – Interface segregation principle
• D – Dependency Inversion Principle

1. The font data is compressed, so sites using WOFF will use less bandwidth and will load faster than if they used equivalent uncompressed TrueType or OpenType files.
2. Many font vendors that are unwilling to license their TrueType or OpenType format fonts for use on the web will license WOFF format fonts. This improves availability of fonts to site designers.
3. Both proprietary and free-software browser vendors like the WOFF format, so it has the potential of becoming a truly universal, interoperable font format for the web, unlike other current font formats.

Every <h1> tag is a note title

Every <h1> tag is preceded by a <a name="#SomeUniqueNumber"/> tag

Note title's, note creation date/tags, and note contents are node siblings a.k.a not nested within one another.

<script type="text/javascript" src="http://your.cdn.com/first.js"></script>

<script type="text/javascript">

<script type="text/javascript" src="https://code.jquery.com/jquery-2.1.1.min.js"></script>

<script type="text/javascript" src="js/materialize.min.js"></script>

<div align="right">

  This is some text!

</div>

img {
    max-width: 100%;
    max-height: 100%;
}

This seems like an outdated video but manages to useful for becoming more familiar with Dreamweaver.

TechnicalCafe shows how to make a multi-page website using notepad++.

To parse an XML document, you can use xmlInternalTreeParse() or xmlTreeParse() (with useInternalNodes specified as TRUE or FALSE) or xmlEventParse(). If you are dealing with HTML content which is frequently malformed (i.e. nodes not terminated, attributes not quoted, etc.), you can use htmlTreeParse(). You can give these functions the name of a file, a URL (HTTP or FTP) or XML text that you have previously created or read from a file.

xmlfile=xmlParse("fileName.xml")

length(xmlChildren(node))

xmlSize(node)

XPath is an XML technology that provides a language for accessing subsets of an XML tree. It allows us to express things such as "find me all nodes named a" or "find me all nodes name a that have no attribute named b" or "nodes a that have an attribute b equal to 'bob'" or "find me all nodes a which have c as an ancestor node". It has a similar feeling to R's subsetting capabilities and works for trees rather than vectors and data frames. It is also very powerful and efficient. But it takes a little time to learn. Some decent tutorials are available on the Web (e.g. Zvon and w3schools) and there are books that cover this subject, e.g. [XML in a Nutshell], [XPathXPointer].

The XPath functions in the XML package are getNodeSet() and xpathApply() . Basically, you specify the document returned from xmlInternalTreeParse() and the XPath expression to identify the nodes. getNodeSet() returns a list of the matching nodes. xpathApply() is used to apply a function to each of those nodes, e.g. find nodes named "a anywhere in the tree that have an "href" attribute and get the value of that attribute

src = xpathApply(doc, "//a[@href]", xmlGetAttr, "href")

Of course, once we have the nodes of interest, we need to be able to extract their information. There are several functions to do this: xmlName() , xmlAttrs() , xmlGetAttr() , xmlChildren() and xmlValue() . xmlName() gets the name of the node/element. xmlAttrs() returns all the attribute name-value pairs as a character vector while xmlGetAttr() is used to query the value of a single attribute with facilities for providing a default value if it is not present and converting it if it is. We tend to use xmlGetAttr() as we typically know which attributes we are looking for. xmlAttrs() is used when doing general/meta- computations.

html.aDates<-xpathApply(doc, "//table[@bgcolor]", xmlValue)

html.aDates<-xpathApply(doc, "//table[@bgcolor]/tr/td/i", xmlValue)

diff<-anchs[is.na(match(anchs,truth)),]

> diff

[[1]]

[1] "tex2html11"

html.titles<-xpathApply(enHTML, "//h1", xmlValue)

This recursively applies the specified function to each node in an XML tree, creating a new tree, parallel to the original input tree. Each element in the new tree is the return value obtained from invoking the specified function on the corresponding element of the original tree. The order in which the function is recursively applied is "bottom-up". In other words, function is first applied to each of the children nodes first and then to the parent node containing the newly computed results for the children. 

                  xmlTreeParse(i, useInternalNodes = TRUE, options = HUGE)
                

html.titles<-xpathApply(enHTML, "//h1", xmlValue)

<h1>What is the &lt;code&gt; tag? It looks like it can do cool things.</h1>

a logical value indicating whether to call the converter functions with objects of class XMLInternalNode rather than XMLNode. This should make things faster as we do not convert the contents of the internal nodes to R explicit objects. Also, it allows one to access the parent and ancestor nodes. However, since the objects refer to volatile C-level objects, one cannot store these nodes for use in further computations within R. They “disappear” after the processing the XML document is completed. If this argument is TRUE and no handlers are provided, the return value is a reference to the internal C-level document pointer. This can be used to do post-processing via XPath expressions using getNodeSet. This is ignored when parsing an HTML document.

html.notes<-xpathApply(enHTML, "//div[not(table[@bgcolor='#D4DDE5'] or ancestor::div)]")

html.tAnchors

html.iLinks

html.titles

The index of every title in html.titles that corresponds to an intra-notebook link is the index of every element in html.tAnchors that equals an element in html.iLinks

iLinkMap<-which(!is.na(match(html.tAnchors,gsub("#","",html.iLinks))))

                      
                        <script type="text/javascript" charset="UTF-8">
    $(function() {
        var plotdiv = document.getElementById("plot1");
        new Dygraph(plotdiv,csvdata,{});
        plotdiv = document.getElementById("plot2");
        new Dygraph(plotdiv,csvdata, {});
        return true;
    })</script>
                      
                    

fDates<-strptime(html.Dates, "%m/%d/%Y %I:%M %p")

sec 0–61: seconds.

min 0–59: minutes.

hour 0–23: hours.

mday 1–31: day of the month

mon 0–11: months after the first of the year.

year years since 1900.

wday 0–6 day of the week, starting on Sunday.

yday 0–365: day of the year.

isdst Daylight Saving Time flag. Positive if in force, zero if not, negative if unknown.

zone (Optional.) The abbreviation for the time zone in force at that time: "" if unknown (but "" might also be used for UTC).

gmtoff (Optional.) The offset in seconds from GMT: positive values are East of the meridian. Usually NA if unknown, but 0 could mean unknown.

acum<-data.frame(dates=fDates, cumsum=1:length(fDates))

html.Dates[which(duplicated(html.Dates))]

[[1]]

[1] "2/17/2015 1:13 PM"

[[2]]

[1] "3/11/2015 6:11 PM"

[[3]]

[1] "3/21/2015 12:13 PM"

[[4]]

[1] "3/21/2015 8:56 PM"

[[5]]

[1] "3/25/2015 10:27 PM"

[[6]]

[1] "3/26/2015 8:37 PM"

Turns out there is a function to handle this in the xts package called make.time.unique()

ds <- options(digits.secs=6) # so we can see the change

x <- xts(1:10, as.POSIXct("2011-01-21") + c(1,1,1,2:8)/1e3)

x

make.index.unique(x)

options(ds)

a[,1]<-1:length(html.Dates)

              
                Per the Range Selector documentation add the highlighted argument below
              
            

                dygraph(nhtemp, main = "New Haven Temperatures") %>% 

              

                  dyRangeSelector(dateWindow = c("1920-01-01", "1960-01-01")) 

              

names(a)[1]<-"changed name"

<iframe width="100%" height="480" frameborder="0" seamless="seamless" scrolling="no" src="path/to/dygraph.html" ></iframe>  

background-color: #b0e0e6;

Review notes for mistakes/missing content

Make Home page "Full Billboard" dygraph

Make annual average uniqueness dygraph

Link pages together

Setup Github repo for Share page

Create 'Bag-o-words' download

Post "This is an intermediate result - do not consider final" disclaimer on analysis

Perhaps post a countdown to when I believe I will be able to post the final results

<div class="collapsible-header"><i class="mdi-navigation">1</i>Third</div>

<span class="numbering" >22</span>

CSS

.numbering {
      text-align: center;
      margin-right: 1rem;
      width: 2rem;
      display: inline-block;
}