Recent questions tagged Hadoop

0 votes

465 views

1 answer

hadoop - Is it possible to append to HDFS file from multiple clients in parallel?

Basically whole question is in the title. I'm wondering if it's possible to append to file located on HDFS from ... 't seems to work. Thanks, See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

528 views

1 answer

hadoop - Forward fill missing values in Spark/Python

I am attempting to fill in missing values in my Spark dataframe with the previous non-null value (if it ... Hadoop suite of tools? Thanks! See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

589 views

1 answer

hadoop - How does Hive decide when to use map reduce and when not to?

As a simple example, select * from tablename; DOES NOT kick in map reduce, while select count(*) from tablename; ... use map reduce (by hive)? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

390 views

1 answer

hadoop - Requests hang when using Hiveserver2 Thrift Java client

This is a follow up question to this question where I ask what the Hiveserver 2 thrift java client API is. ... 3 would help narrow my search. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

639 views

1 answer

hadoop - Finding directories older than N days in HDFS

Can hadoop fs -ls be used to find all directories older than N days (from the current date)? I am trying to ... days prior to the current date. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

704 views

1 answer

hadoop - How to generate Date Series in HIVE? (Creating table)

Suppose I currently have a table that has 1 row for each account and the data in the tables are: Account Number ... to do that as well? Thanks! See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

706 views

1 answer

hadoop - dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured

I was trying to configure hadoop with one name node and four data nodes. I was able to successfully configure the name ... What do I need to do? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

434 views

1 answer

hadoop - A way to read table data from Mysql to Pig

Everyone know that Pig have supported DBStorage, but they are only supported load results from Pig to mysql like that ... but not success..... See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

587 views

1 answer

hadoop - HDFS file watcher

Can I have file watcher on HDFS? Scenario: The files are landing on HDFS continuously.I want to start a ... help will be appreciated.Thanks. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

734 views

1 answer

hadoop - get "ERROR: Can't get master address from ZooKeeper; znode data == null" when using Hbase shell

I installed Hadoop2.2.0 and Hbase0.98.0 and here is what I do : $ ./bin/start-hbase.sh $ ./bin ... </value> </property> </configuration> See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

447 views

1 answer

hadoop - Google Analytics - Getting raw data logs

I have an app that sends data to Google Analytics. I am interested in accessing and storing this data ... /thoughts/suggestions are appreciated. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

864 views

1 answer

hadoop yarn - How to limit the number of retries on Spark job failure?

We are running a Spark job via spark-submit, and I can see that the job will be re-submitted in the ... "GC overhead limit exceeded" issue. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

430 views

1 answer

hadoop - Spark + Scala transformations, immutability & memory consumption overheads

I have gone through some videos in Youtube regarding Spark architecture. Even though Lazy evaluation, Resilience of data ... cache () works? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

639 views

1 answer

hadoop - yarn is not honouring yarn.nodemanager.resource.cpu-vcores

I am using Hadoop-2.4.0 and my system configs are 24 cores, 96 GB RAM. I am using following configs mapreduce ... . Am I doing something wrong?? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.4k views

1 answer

hadoop - How to load CSV data with enclosed by double quotes and separated by tab into HIVE table?

I am trying to load data from a csv file in which the values are enclosed by double quotes '"' and tab ... and ignore header of the file? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

556 views

1 answer

hadoop - Reading HDFS and local files in Java

I want to read file paths irrespective of whether they are HDFS or local. Currently, I pass the local paths ... other better way than this? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

519 views

1 answer

hadoop - How to know what is the reason for ClosedChannelExceptions with spark-shell in YARN client mode?

I have been trying to run spark-shell in YARN ... scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply(YarnScheduler...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

496 views

1 answer

hadoop - Is caching the only advantage of spark over map-reduce?

I have started to learn about Apache Spark and am very impressed by the framework. Although one thing which keeps ... educates me more on spark. See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

569 views

1 answer

hadoop - Spark RDD - is partition(s) always in RAM?

We all know Spark does the computation in memory. I am just curious on followings. If I create 10 RDD in my ... size, where will data to stored? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

496 views

1 answer

hadoop 2.2.0 64-bit installing but cannot start

I am trying to install Hadoop 2.2.0 Cluster on the servers. For now all the servers are 64-bit, I download ... , what do the other errors mean? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

486 views

1 answer

hadoop - SparkSQL vs Hive on Spark - Difference and pros and cons?

SparkSQL CLI internally uses HiveQL and in case Hive on spark(HIVE-7292) , hive uses spark as backend engine. ... and cons of both approaches? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

594 views

1 answer

hadoop - java.net.URISyntaxException when starting HIVE

I am new in HIVE. I have already set up hadoop and it works well, and I want to set up Hive. ... :user.name%7D Are there any solutions? See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

530 views

1 answer

hadoop - PIG how to count a number of rows in alias

I did something like this to count the number of rows in an alias in PIG: logs = LOAD 'log' logs_w_one = ... me if there is a better way! See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

740 views

1 answer

hadoop copy a local file system folder to HDFS

I need to copy a folder from local file system to HDFS. I could not find any example of moving a folder( ... /Source-Folder-To-Copy HDFS-URI See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

496 views

1 answer

hadoop - No data nodes are started

I am trying to setup Hadoop version 0.20.203.0 in a pseudo distributed configuration using the following guide: http://www. ... *************/ See Question&Answers more detail:os...

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

647 views

1 answer

hadoop - PIG how to count a number of rows in alias

I did something like this to count the number of rows in an alias in PIG: logs = LOAD 'log' logs_w_one = ... me if there is a better way! See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

579 views

1 answer

hadoop - Where does Hive store files in HDFS?

I'd like to know how to find the mapping between Hive tables and the actual HDFS files (or rather, directories ... Hive store its files in HDFS? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

438 views

1 answer

hadoop - Avro vs. Parquet

I'm planning to use one of the hadoop file format for my hadoop related project. I understand parquet is efficient ... to me in simple terms? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags