Recent questions tagged Hadoop

0 votes

604 views

1 answer

My map is currently inefficient when parsing one particular set of files (a total of 2 TB). I'd like to ... as copying from local to dfs.) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

377 views

1 answer

hadoop - What is Google's Dremel? How is it different from Mapreduce?

Google's Dremel is described here. What's the difference between Dremel and Mapreduce? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

482 views

1 answer

hadoop - Save Spark dataframe as dynamic partitioned table in Hive

I have a sample application working to read from csv files into a dataframe. The dataframe can be stored to a ... any help is much appreciated. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

725 views

1 answer

hadoop - When using --negotiate with curl, is a keytab file required?

The documentation describing how to connect to a kerberos secured endpoint shows the following: curl -i --negotiate ... command be looking for? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

527 views

1 answer

hadoop - Hive installation issues: Hive metastore database is not initialized

I tried to install hive on a raspberry pi 2. I installed Hive by uncompress zipped Hive package and ... with Derby would be appreciated! See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

881 views

1 answer

hadoop - Job queue for Hive action in oozie

I have a oozie workflow. I am submitting all the hive actions with <name>mapred.job.queue.name</name> < ... behind this behavior and solution. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

758 views

1 answer

hadoop - Hive: Add partitions for existing folder structure

I have a folder structure in HDFS like below. However, no partitions were actually created on the table using the ... csv /file0001.csv ...etc See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

817 views

1 answer

hadoop - Sqoop - Data splitting

Sqoop able to import data from multiple tables using --query clause but not clear whether it is able to import ... the priority? Thanks.... See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

738 views

1 answer

hadoop - Access a secured Hive when running Spark in an unsecured YARN cluster

We have two cloudera 5.7.1 clusters, one secured using Kerberos and one unsecured. Is it possible to run Spark ... we can't achieve the above. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

650 views

1 answer

hadoop - Create HIVE Table with multi character delimiter

I want to create a HIVE Table with multi string character as a delimiter such as CREATE EXTERNAL TABlE tableex(id ... multi string like "~*". See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

829 views

1 answer

hadoop - java.sql.SQLException: org.apache.thrift.transport.TTransportException in hive?

I'm trying to create table in hive with java. I found java.sql.SQLException: org.apache.thrift.transport.TTransportException ... can i fix it.? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

785 views

1 answer

hadoop - Is it better to have one large parquet file or lots of smaller parquet files?

I understand hdfs will split files into something like 64mb chunks. We have data coming in streaming and we can ... having, say, 1gb files? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

594 views

1 answer

hadoop - Where are logs in Spark on YARN?

I'm new to spark. Now I can run spark 0.9.1 on yarn (2.0.0-cdh4.2.1). But there is no log ... 2> $LOG_DIR/stderr Where can LOG_DIR be set? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

586 views

1 answer

hadoop - Loading Data from a .txt file to Table Stored as ORC in Hive

I have a data file which is in .txt format. I am using the file to load data into Hive tables. When I load ... to store data into an ORC table? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

396 views

1 answer

hadoop - Pig Latin: Load multiple files from a date range (part of the directory structure)

I have the following scenario- Pig version used 0.70 Sample HDFS directory structure: /user/training/test/ ... comma separated list? cheers See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

614 views

1 answer

hadoop - Spark iterate HDFS directory

I have a directory of directories on HDFS, and I want to iterate over the directories. Is there any ... Spark using the SparkContext object? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

578 views

1 answer

hadoop - Hbase quickly count number of rows

Right now I implement row count over ResultScanner like this for (Result rs = scanner.next(); rs != null; ... to quickly count number of rows. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

408 views

1 answer

hadoop - Easiest way to install Python dependencies on Spark executor nodes?

I understand that you can send individual files as dependencies with Python Spark programs. But what about full ... number of distributed nodes? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

568 views

1 answer

hadoop yarn - Which cluster type should I choose for Spark?

I am new to Apache Spark, and I just learned that Spark supports three types of cluster: Standalone - meaning ... cluster type should I choose? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

558 views

1 answer

hadoop - Hive cluster by vs order by vs sort by

As far as I understand; sort by only sorts with in the reducer order by orders things globally but shoves ... doesn't always do that. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

440 views

1 answer

hadoop - Apache Spark: The number of cores vs. the number of executors

I'm trying to understand the relationship of the number of cores and the number of executors when running a ... How to explain this result? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

420 views

1 answer

hadoop - How to update table in Hive 0.13?

My Hive version is 0.13. I have two tables, table_1 and table_2 table_1 contains: customer_id | items | price | ... using join, but it fails. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

487 views

1 answer

hadoop - Hive unable to manually set number of reducers

I have the following hive query: select count(distinct id) as total from mytable; which automatically spawns: 1408 ... in more reducers? Thanks! See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

885 views

1 answer

hadoop - Pyspark: get list of files/directories on HDFS path

As per title. I'm aware of textFile but, as the name suggests, it works only on text files. I would need to ... a local path. I'm using pyspark. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

473 views

1 answer

hadoop - OOZIE: properties defined in file referenced in global job-xml not visible in workflow.xml

I'm new to hadoop and now I'm testing simple workflow with just single sqoop action. It works if I use ... /value> </property> </configuration> See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

640 views

1 answer

hadoop yarn - Apache Spark: setting executor instances does not change the executors

I have an Apache Spark application running on a YARN cluster (spark has 3 nodes on this cluster) on cluster mode. ... is 12g in yarn-site.xml See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

584 views

1 answer

hadoop - Spark on yarn concept understanding

I am trying to understand how spark runs on YARN cluster/client. I have the following question in my mind. Is ... is sending the job to cluster? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

495 views

1 answer

hadoop - Namenode not getting started

I was using Hadoop in a pseudo-distributed mode and everything was working fine. But then I had to restart my ... sh and bin/start-dfs.sh. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags