R语言学习笔记之

Standalone模式：Standalone模式运行的Spark集群对不同的应用程序采用先进先出（FIFO）的顺序进行调度。默认情况下每个应用程序会独占所有可用节点的资源。

现在版本的SparkR只能运行在standalone模式下

问题1：安装问题

由于R涉及到Fortran语言，要下载gcc-gfortran包

安装步骤：1)将R-3.2.3.tar.gz解压 2)./configure 3)make 4)make install(这步可以没有) 5)配置环境变量 vi .bash_profile

./configure的时候会出现以下错误：

--with-readline=yes (default) and headers/libs are not available 这是由于需要依赖readline-devel包的缘故 yum install readline-devel即可

configure: error: cannot compile a simple Fortran program 这是由于需要依赖gcc-gfortran包的缘故 yum install gcc-gfortran即可

configure: error: --with-x=yes (default) and X11 headers/libs are not available 这是由于需要依赖libXt-devel包的缘故 yum install libXt-devel即可

以上步骤依赖了较多的包：①gcc ②gcc-c++ ③readline-devel ④gcc-gfortran ⑤libXt-devel

[plain]view plaincopy

1. yum install libXt-devel
2. yum install readline-devel

[plain]view plaincopy

1. yum install gcc
2. yum install gcc-c++
3. yum install gcc-gfortran
4. tar -zxvf R-3.2.3.tar.gz
5. cd R-3.2.3
6. ./configure
7. make

问题2：

unsupported URL scheme
Warning: unable to access index for repository https://rweb.crmda.ku.edu/cran/src/contrib

镜像问题，解决方式有两种：1）换镜像，即在选择的时候改 2）install.packages("RODBC", dependencies = TRUE, repos = "http://cran.rstudio.com/")

问题3：在安装R包的时候遇见错误

configure: error: "ODBC headers sql.hand sqlext.h not found"

是因为没有在Linux 下安装ODBC包。RODBC 需要 unixODBC 和unixODBC development 包，使用YUM 安装之后即可解决。

yum install unixODBC

yum install unixODBC-devel

则之后再install.packages("RODBC", dependencies = TRUE, repos = "http://cran.rstudio.com/")

一直连不上远程数据库，要查看一下是不是网络不通，ping一下远程主机。

SparkR编程示例：

[plain]view plaincopy

1. #如果直接调用的sparkR,则不用设置Sys.setenv和.libPaths，直接library(SparkR)即可

[plain]view plaincopy

1. #Sys.setenv(SPARK_HOME = "D:/StudySoftWare/Spark/spark-1.5.2-bin-hadoop2.6")
2. #.libPaths(c(file.path(Sys.getenv("SPARK_HOME"),"R","lib"), .libPaths()))
3. library(SparkR)
4. sc <- sparkR.init(master = "local")

[plain]view plaincopy

1. #sc <- sparkR.init(master = "spark://192.168.133.11:7077") 以集群方式运行
2. sqlContext <- sparkRSQL.init(sc)
3. DF <- createDataFrame(sqlContext, faithful)
4. head(DF)
5. localDF <- data.frame(name=c("John", "Smith", "Sarah"), age=c(19, 23, 18))
6. df <- createDataFrame(sqlContext, localDF)
7. # Print its schema
8. printSchema(df)
9. # root
10. # |-- name: string (nullable = true)
11. # |-- age: double (nullable = true)
12. # Create a DataFrame from a JSON file
13. path <- file.path(Sys.getenv("SPARK_HOME"), "examples/src/main/resources/people.json")
14. peopleDF <- jsonFile(sqlContext, path)
15. printSchema(peopleDF)
16. # Register this DataFrame as a table.
17. registerTempTable(peopleDF, "people")
18. # SQL statements can be run by using the sql methods provided by sqlContext
19. teenagers <- sql(sqlContext, "SELECT name FROM people WHERE age >= 13 AND age <= 19")
20. # Call collect to get a local data.frame
21. teenagersLocalDF <- collect(teenagers)
22. # Print the teenagers in our dataset
23. print(teenagersLocalDF)
24. # Stop the SparkContext now
25. sparkR.stop()

Java.io.IOException: Cannot run program "Rscript": error=2, No such file or directory 遇到这种错误是因为：

looks like the issue was that code was looking for Rscript under "/usr/bin". Our default installation was /usr/revolutionr.

Just created a link Rscript in /usr/bin that points to /usr/revolution/bin/Revoscript

或者拷贝一份Rscript到/usr/bin目录下即可,参考：https://github.com/RevolutionAnalytics/RHadoop/issues/87

示例二：wordCount

[plain]view plaincopy

1. library(SparkR)
2. sparkR.stop()
3. #调用sparkR的时候会自动的初始化一个SparkContext，默认是local模式
4. sc <- sparkR.init(master="spark://<pre name="code" class="plain">192.168.133.11

:7077","WordCount")#sparkR.init(master = "", appName = "SparkR",sparkHome = Sys.getenv("SPARK_HOME"), sparkEnvir = list(),sparkExecutorEnv = list(), s#parkJars = "", sparkPackages = "")

[plain]view plaincopy

1. lines <- SparkR:::textFile(sc, "hdfs://namenode主机名/user/root/test/word.txt")

[plain]view plaincopy

1. words <- SparkR:::flatMap(lines, function(line) { strsplit(line, " ")[[1]] })

[plain]view plaincopy

1. wordCount <- SparkR:::lapply(words, function(word) { list(word, 1L) })

[plain]view plaincopy

1. counts <- SparkR:::reduceByKey(wordCount, "+", 2L)

[plain]view plaincopy

1. #如果要保存到hdfs中，则path要写成"hdfs://namenode主机名/user/root/test/sparkR.txt") path要给出全路径

[plain]view plaincopy

1. SparkR:::saveAsTextFile(counts, "hdfs://namenode主机名/user/root/test/sparkR.txt")

2. ##如果要保存到hdfs中，则path要写成"hdfs://namenode主机名/user/root/test/sparkR.txt") path要给出全路径

   ##如果要将createDataFrame(hc,生成的 sparkr dataframe 以文件形式存到hive中 需要先将其转为rdd

   data_in_rdd <- SparkR:::toRDD(data_in)

   SparkR:::saveAsTextFile(data_in_rdd, evo_table_name_lower_with_path)

[plain]view plaincopy

1. output <- SparkR:::collect(counts)

API documentation1：http://amplab-extras.github.io/SparkR-pkg/rdocs/1.2/index.html，该网址给出的API要这样调用SparkR:::函数名

API documentation2：http://spark.apache.org/docs/1.5.2/api/R/index.html，该网址给出的API可以直接调用。